Did you ever spend a week wading through sustainability disclosures trying to get to the bottom of niche but vital questions of our time like comparing the profitability of major oil and gas producers with the amount of greenhouse gas emissions that could be attributable to them (a lot)?

I have. Fun it was not.

I emerged more informed about our world, weary in the way that you tend to feel after spending extensive effort to find information that, ultimately, wasn't long on cause for optimism.

But I also left wondering why it seemed to take exhaustive acts of research, and inordinate perseverance, to answer what should really be a simple question that should be answerable quickly from public-source information.

Thanks to the powers of the internet put on steroids by the information-mining power of tools like ChatGPT, anyone motivated to find out the current temperature in Beijing or the approximate cost of a pizza in London can do so in about five seconds.

But try to answer this question and you're in for a bumpier ride:

"What if we added together the reported GHG emissions for Shell in 2023 and attempted to convert them into dollars at one of the various suggested social cost of carbon numbers? And what if we compared that number to their [financial metric]?"

Reading this literature is a bit like stepping into a strange alter universe in which you find yourself wondering whether they were written page by page by entirely different people.

You might find yourself scratching your head in confusion: isn't performance here actually getting worse? If so why is that the headline, then? It's mildly crazy-inducing in a way that it almost shouldn't be.

While some companies have made notable efforts to make these data-analysis friendly, many, if not the vast majority, have not. Those inclined towards endorsing conspiracy theories would likely retort: "yes Daniel, that's the entire point. So that you don't read these things."

Call me naive, but I try to take a slightly more benevolent view. I think that many mean well but overestimate the ability of an interested readership to actually glean useful information from their content. The norm: a long and very glossy PDF. Great for investors and determined sector analysts zoned in on the pivots up and down in the line chart. For the great unwashed rest of us? Much harder to process.

Or this one from BP (also 2023):

Notice also how easy it is to miss the fact that BP's reported scope 3 emissions (for 2023) at 315 MtCO2) were actually their second year-on-year increase since 2021:

Finally, let's get down to the technical brass tacks of how this proof of concept actually worked:

Firstly, I validated this as a Langchain script writing out the reports as markdown files and as a chatbot (frontend: OpenAI).

Then, I tried a version which asks for (the ultimate objective) gathering the data of five companies and comparing them against one another. However, this increases drastically the context load on the LLM. Not only does each datapoint need to be gathered five times over and held within one context window, all the sources need to be gathered too.

As long context models and better hardware break down the context challenge, this will probably be less of a problem over time. But right now, and while hugely exciting, the AI at our disposal requires a bit of hackery to really bring its capabilities to life.

Nevertheless, beginning from a modest starting point (gather three datapoints, multiply them by one multiplier, format them in *one format) the results have sprung cause for optimism, which is something I left short on when reading them by hand.

What's also been impressive as I have analysed and validated the results: not only can the models pick out the right datapoints (no small feat when even the units of measurement are inherently confusing, like millions of metric tons of carbon dioxide equivalents), they can perform accurate mathematics, including when the other units are denoted in notation like $23.4M (determining 'M' to be shorthand for millions).

Tools That Make This Work (Data Gathering, Analysis, And Visualisation Stack)

While it's tempting to attempt to get an LLM to do literally all of the analysis and visualisation workload asked for here (believe me, I tried!), it's probably both more realistic and responsible to try to divvy up the task between services.

But the core of the stack here - and why I remain hugely optimistic about their role in reshaping all manner of tech workflows - is the large language model (LLM).

A modern LLM might be using RAG to pull in the data and an external API to do the actual number crunching, but it's still absolutely indispensable to the overall project.

AI workflows might be becoming increasingly complex and leveraging increasingly complicated arrays of tools, but the LLM is still the orchestrator providing the pivotal bridge between the world of human language and the technical universe within which it sits.

As I discovered through trial and error, not just any LLM will do.

For one, this is an instructional use case and not much of a conversational one - so choose an instructional model.

It demands reasoning, analytical abilities, and an understanding of computations - so choose a model or variant that plays to those strengths (despite its high API pricing, OpenAI's o1-mini ticks most of these boxes).

Having knowledge of these sustainability disclosures would obviate the need for RAG entirely - so a model with a recent training data cutoff would be another natural fit.

Accessory modules:

• Wolfram Alpha is an outstanding add-on for computation.


• The list of promising data visualisation tools is a bit shorter, but this probably stretches past the part of this process that it makes any sense to do by AI anyway.

Once you've assembled this stack, you've theoretically got:

• Something to gather the data


• Something to parse the data


• Something to visualise the data

Then scale all of this up to:

• Gather & parse reports at scale
  


• Parse and analyse data at scale


• Generate insights and comparisons at scale

Variations & Config Notes

The configuration text is below, but here are a few notes about some of the deliberate choice of language it contains:

Hallucination prevention is key to this kind of workload. Although my intention in developing this was creating a model to frontload the task of human review (hence the demand for sources), you still want to include robust language to break up a chain if data can't be found like:

Only use X as a data source. If you cannot retrieve the sustainability data for this financial year, inform the use that you cannot complete the task.

Other slightly more passive-aggressive instructions can be added on for further insistence like:

Do not return speculative information under any circumstance.

The checkpoint that I ultimately baked into my script was this one:

If you are certain that you can retrieve and parse the requested data for {company} continue. If you cannot, inform the user that you cannot reliably perform the requested analysis.

Output formatting

Another thing I love about prompt engineering and why working with information via an LLM is (to my mind) so much more engrossing than doing so with a search engine: the ability to shape how you want to receive the information retrieved.

In this case - and when working with LLMs generally - adding verbiage to instruct the LLM to favor a specific output format is the go-to technique.

The output format can be markdown (if using this as a script; I got mine to fill up .md files for every company within a Github repository).

Or you can configure an OpenAI Assistant to provide JSON for easier intake into analytics and automation workloads.

In OpenAI Platform, change the response format selector:

Output is a formatted analysis using the requested calculations (reported carbon dioxide equivalents emitted simulated 'monetisation' at the $236/mtco2e rate proposed by the International Foundation For Valuing Impacts):

Validated: EBITDA:

End Data Visualisation Chain

The flexibility in this approach for first-pass data analysis and visualisation lies in the enormous flexibility provided by asking large language models (LLMs) to format their output according to specific instructions.

For example, for markdown, with PDF as the ultimate target:

Enclose the output within a codeblock, formatted as markdown

For CSV, with ingestion to a database/analytics tool/date warehouse as the objective:

Output the gathered data as raw CSV enclosed within a codeblock

To achieve a consistent output format on repeated/batch runs, a standard header row can be written and then passed as an instruction:

Output the gathered data as raw CSV enclosed within a codeblock. Adhere precisely to this format.

You can even request that the header row be included to validate that the format instruction has been followed and then strip from the output if it has:

Output the gathered data as raw CSV enclosed within a codeblock. Return the header row (row 1) and then your findings (row 2).