Deneb Annual Round-Up

It's a wonderful time of the year when I look over community contributions for my membership in the Microsoft MVP program. As some of this includes looking at how Deneb is doing, I thought it'd also be a good time to provide an update on Deneb's usage and a few bits about what's been happening lately.

Looking Back

When I started figuring out whether Deneb would be possible or feasible, it was probably mid-2020. Once some things had been proven, things started to ramp up in late 2020, with us starting the public preview in April 2021. By September, we'd made the AppSource submission, with certification being achieved in November 2021.

I couldn't realistically determine whether building visuals with a tool like Deneb would be acceptable to folks. I've long been a fan of the Vega and Vega-Lite languages, and bringing them into Power BI was always a question of how well received they might be - the languages are powerful, but they aren't the "Power BI way" of doing things. However, it solved real problems for me, so I've been more than happy to make it a thing and keep maintaining it.

Deneb was (and is) intended as a free and open-source visual with community support. I've been clear from the beginning that community adoption and collaboration are needed for any long-term viability. We had some great people out of the gate who "got it" and were happy to share what they were learning and could achieve, which was fantastic to see. Many of these examples are still available (and being added to by their authors) via the community page if you want to check them out and show these folks some love and appreciation. I'm eternally grateful for their continued support. And many thanks as ever to our community experts who volunteer to be contacted for questions, allowing me to focus on development and maintenance, as well as communities like r/PowerBI and the Fabric Community forums, where many of you have shared your experiences and examples, or make others aware of Deneb as a solution.

Where We've Come to (Usage-Wise)

But back to that viability thing. This time last year, I was reviewing Deneb's usage for the (MVP) year and was blown away to see that we had around ~75,000 downloads by developers over ~10,000 organizations. These are downloads to reports by authors; unfortunately, it's impossible to know how many end users are seeing and using these visuals as part of their analytics platform as Deneb is a certified visual (meaning that we can't do telemetry to find this stuff out). Everything I know about usage is what people have been gracious enough to tell me about, whether it's sharing examples, recommendations to others that I happen to come across, or through reviews on AppSource (28 5-star reviews and climbing!)

So, when it came to doing the same review this year, I thought it would be great if we could sustain the same rate of downloads over the year, but we kind of smashed that. As of writing, we got nearly 113,000 downloads from over 16,000 organizations, an increase of ~48% over the previous review period 🎉

This is great for free and open-source data visualization and continued adoption for the Vega ecosystem within Power BI. It's also fantastic news that Vega projects have recently been included in the projects sponsored by NumFOCUS, and new releases of each language are coming soon!

Version 1.7 of Deneb (released last year) is very close to my original vision for getting things working in Power BI at the start of my journey. However, we've learned a lot since then, and our users have had great ideas via our GitHub repo's issue tracker.

Some of these have been delivered. Some are possible but not easy to achieve using the current architecture, and some are great feature ideas but won't be possible due to limitations with the custom visual ecosystem, which is managed and maintained by Microsoft. I have been incredibly unlucky in getting anything done here, but I will continue to lobby the team for whatever I can get, and if things ever change, maybe we can unlock some of these features.

Since then, we've also had features and tools appear in Power BI, like field parameters and PBIR, which extend the functionality of visuals and the source control capabilities of reports accordingly. The "v1" Deneb template format (a Vega or Vega-Lite spec and some basic metadata) is simplistic and it's challenging to extend via the visual properties system without significant burden. If folks are keen to start doing things with PBIR and Deneb, it's not intuitive under this current paradigm.

So, What's Next?

I want Deneb to be around for many, many more years and to continue to be a free and open-source visual for the community. However, for this to work, we need a better platform to build upon. Therefore, I've been planning, proofing, and developing for the last 6 months on what the future looks like. If I can keep the current work rate up, I'll have more solid details (and demonstrable features) in a few months and an ETA for when this version will start to become more visible.

But here are some things that you can rely upon in terms of what to know going forward at this stage:

Deneb 1.x

• The current v1 Deneb will remain in AppSource and certified in perpetuity, so your visuals and reports will continue to work.
• I'll keep this up to date regarding bugs, security, and custom visual API updates to make it as future-proof as possible for the current feature set.
• I'll keep the Vega and Vega-Lite languages as current as possible for each of the current "v5" releases. If future major versions of the languages affect backward compatibility, these will be frozen as of v5.
• The status quo will remain regarding v1 being free and open-source, with community (MIT) support.

Deneb 2.x

First things first

With the last bullet point for v1, I may have subconsciously primed you for some expectations around a major version with commercial changes. First, I'll nip that in the bud and say that Deneb "v2" is a major version change for architectural reasons; it's a complete re-write based on all the lessons learned. The Power BI visual will remain free, open-source (and MIT licensed).

• The largest change is that the property/persistence system that integrates with Power BI property storage will be very different, more centralized, and extensible for new features. It will have a dedicated JSON schema, allowing developers to validate changes outside of a report via PBIR.
• These changes mean that it will be a separate Power BI visual in AppSource with a different unique ID and will be certified. This also means that any visuals in your reports will remain v1 unless you replace them with the new visual.
• It's not going to be possible to swap a v1 visual to a v2 visual in place (like you might be able to do for some of the core visuals). We don't have low-level access to migrate properties over. As we want to significantly clean up the visual metadata, the older v1 properties to manage the state will be removed.
• The likely migration path will be to export the current visual as a v1 template, which you can import into a v2 visual. It'll handle the conversion for you, and you can pick up where you left off. I'd also like this process to be available as an API for anyone wishing to migrate visuals using scripting and PBIR, but that will depend on complexity and free time (I will do my best to fit it in, though).

In terms of new features, the following are confirmed and more or less implemented in the current v2 builds:

• Linked data (dataset) will be applied to the Vega view via streaming rather than re-processing everything. The re-processing that v1 does causes a complete re-render, and this streaming approach prevents the view from resetting when canvas events (like cross-filtering) occur and should open up many more use cases for dynamic and actionable visuals.
• Field parameters will be supported by their name rather than by the internal column or measure they are impersonating. We've had to get some changes provided by MS in conjunction with some significant architectural changes, but there will be a way of managing them (original proposal here, which is changing a bit but is fundamentally a good indicator of how they will be available to your dataset). If preferable, we'll also offically support the 'legacy' method (where folks have worked around the limitations through creative semantic modeling) as much as we can.
• There will be support for drill-down. At the bare minimum, it will be at the dataset level (operations from the visual header affecting the query against the visual).
• Pick and choose which support columns you need (e.g., formatting, cross-highlighting) rather than having everything in your dataset.
• Improvement around the display and UX for bindable signals.

As mentioned above, there will be much more to include in this list as we progress. I hope that at least some of these will pique your interest for now.

Thanks once again to everyone for your continued support, and I'm looking forward to the next year and beyond!

All the best,

DM-P