Skip to content

Add write support to netidx-excel#3

Open
base1172 wants to merge 10 commits intonetidx:masterfrom
base1172:master
Open

Add write support to netidx-excel#3
base1172 wants to merge 10 commits intonetidx:masterfrom
base1172:master

Conversation

@base1172
Copy link
Copy Markdown

This PR makes several changes:

  • Adds support for writing to netidx paths via the =NetSet() UDF
  • Adds =NetGet() UDF
    • This is just for convenience; under the hood its behavior is functionally identical to Excel's built-in =RTD()
  • Automatically registers the COM DLL via xlAutoOpen
    • Yay, users don't need to manually run regsvr32 anymore!
  • Registers the COM DLL in the per-user hive instead of in HKEY_CLASSES_ROOT
    • Yay, users don't need admin privileges to register the COM DLL anymore!
  • Changes the RTD server behavior so that it initially returns #GETTING_DATA instead of the integer 1 when connecting to data
    • #GETTING_DATA is a built-in Excel error. Returning a fully-fledged Excel error (which will propagate as an error to any dependent cells) is preferable to returning a string value such as "#SUBSCRIBING".
  • Fixes a bug in irtd_update_event_thread that could cause its IStream object to be released twice
  • Removes dependence on the once_cell crate (that functionality is now available in std::sync::LazyLock)
  • Removes dependence on the winreg crate.

@base1172
Copy link
Copy Markdown
Author

@eric-stokes-architect Some implementation details to be aware of:

  • Excel doesn't expose a trivial way to know when a UDF is no longer in use, which means that the =NetSet() UDF doesn't have a way to drop subscriptions that are no longer in use. I believe this is a solvable problem, but this PR doesn't try to tackle it. For now, any subscriptions opened by =NetSet() will remain open until the process terminates.
  • The =NetSet() UDF is marked as non-volatile. This means if a workbook is saved and then reopened, cells that use =NetSet() will only write a new value if an input changes. If no inputs change, no value will be written. This can be unintuitive for users, as they probably expect =NetSet() to re-write all values exactly once when the workbook is loaded.
  • All interaction with Excel is done via the Excel XLL SDK. Bindings for the SDK were generated with bindgen and are in src/xll_utils/xlcall.rs. Instead of making this a static file, we could generate the bindings at build time; however, the user would need to have the Excel XLL SDK headers installed to make it work.
  • With this PR, the add-in will now register its COM RTD server automatically when the add-in is loaded by Excel, with registry entries placed in the user hive instead of HKEY_CLASSES_ROOT. From the user's perspective I believe this behavior is strictly better...the add-in should "just work" as soon as it is loaded in Excel, without users needing admin privileges or running regsvr32.

@estokes
Copy link
Copy Markdown
Member

estokes commented Mar 10, 2025 via email

@base1172
Copy link
Copy Markdown
Author

In the initial version of this PR there was a subtle memory leak in the =NetGet() UDF. Apparently, Excel requires that if we have a UDF that calls an Excel callback, and the Excel callback returns a heap-allocated LPXLOPER12, and if our UDF will return that same LPXLOPER12 as the UDF's return value, then we must set the xlbitXLFree on the LPXLOPER12 before returning it.

I've updated the PR to fix that issue and manually confirmed that the new implementation no longer leaks.

For background on xlbitXLFree and Excel memory management, see https://learn.microsoft.com/en-us/office/client-developer/excel/memory-management-in-excel#returning-xloperxloper12s-to-be-freed-by-excel

@base1172
Copy link
Copy Markdown
Author

I've disabled registration of the =NetGet() UDF in this PR, as we've had user reports of Excel leaking memory when using =NetGet(). The memory leak goes away when users switch to the built-in =RTD() function.

I don't know why Excel leaks memory when using the =NetGet() RTD wrapper but not when using =RTD(). I've pored over the code in this PR multiple times looking for a cause, but I can't find anything. I have a hunch the leak is a bug in Excel, possibly related to Excel's behavior of caching RTD topics for the application lifetime (details here and here), but I don't have the free cycles right now to confirm if that's the cause.

I plan to revisit this issue at a later date, as using =NetGet() to subscribe to netidx paths is more convenient than =RTD(). Until then, the best option is to just disable =NetGet().

@estokes
Copy link
Copy Markdown
Member

estokes commented Mar 23, 2025

Have you tried wrapping it in VBA? I know that's not an ideal situation, but IIRC it may not leak memory. It may also not perform as well. Don't we just LOVE Excel 😂

Copy link
Copy Markdown
Member

@estokes estokes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a solid piece of work, thanks! I have a couple of small comments, and one bigger suggestion.


```
=RTD("netidxrtd",, PATH)
=NetGet(PATH)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're rolling back wapping in a UDF then fix this doc

"Win32_System_Threading",
"Win32_Security",
"Win32_Security_Authorization", ] }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, thank you for updating this. Microsoft pretty much rewrites the public API of the windows crate every time they release a new version.

return 0;
}
_ => (),
}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They seriously switched to Result in 0.48 and then back to HRESULT in 0.60 ...

would this maybe look nicer as

if CoInitialize(None).is_err() {
... error stuff
}

Ok(s) => match NetSetType::try_from(ty) {
Err(()) => XLOper12::error(XlErr::NA).into(),
Ok(typ) => {
let path: netidx::path::Path = Into::<netidx::path::Path>::into(s);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are both annotations really needed?

rt.block_on(async move { Subscriber::new(cfg, desired_auth) })?;
std::thread::Builder::new().name("netidx-setter".into()).spawn(move || {
// TODO: Add support for closing unused [Dval] by integrating with the RTD server. We need to marshal [Dval] ids to/from strings when a new path is subscribed or when an old path is dropped.
// Paths that are already subscribed and still in use won't need to be marshaled.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dvals are automatically shared on the same subscriber, so all you would need to do to make this happen is ensure that you are using the same subscriber for RTD and for write. Then, when you write, just subscribe to the path again, and you'd get back a clone of the existing Dval, do the write, and then drop it when done, the subscription will be kept alive as long as the RTD server is still using that subscription.

There's no easy way we're ever going to be able specify a netidx config for Excel, so at this point I'd support initializing netidx, the async runtime, and the subscriber in a global LazyLock. Then we can share the subscriber everywhere it needs to be shared, and avoid a lot of problems.

after this you don't need a hashmap, you should be able to just call subscriber.subscribe(path).write(value), since everything related to Dvals is ordered this should work even if we were not previously subscribed.

@@ -0,0 +1,128 @@
mod registration;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a solid piece of work, should it be a separate crate? It seems general enough that others may wish to use it.

@estokes estokes self-assigned this Mar 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants