The AI Alliance

The hype cycle

Gartner identified a phenomenon called the Hype Cycle where a technology dazzles everybody then as reality bites, expectations become more reasonable.

Where are we on this cycle for LLMs?

Innovation Trigger

It’s been a year since ChatGPT was released to near universal acclaim. Now things are calming down, let’s strip away the hype and see if it can actually do something useful other than code snippets.

This is a real life problem. We have 650 lines of SQL Server code that we want converted to the Spark SQL dialect. We wonder if ChatGPT can save us tediously converting it by hand.

Here is our experience of the hype cycle but compressed into one day.

Peak of Inflated Expectations

ChatGPT starts by giving a nice summary of what the SQL does. It then converts the first 50 lines or so. These are correct but do not present much of a challenge. Still, so far so good.

Now it gets to its first real challenge. SQL Server and Spark have the parameters of the DATEDIFF keyword in the opposite order to each other. Does ChatGPT know this?

Bravo! It correctly reverses the arguments.

Trough of Disillusionment

Now we encounter the first oddity. ChatGPT hallucinates a table that doesn’t exist.

Tables

• SET_RecordIds_0420
• SET_RecordIds_0520
• SET_RecordIds_0620
• SET_RecordIds_0720
• SET_RecordIds_0820
• SET_RecordIds_0920

exist but SET_RecordIds_1020 is totally fictitious. Yeah, the names of these tables are not great but - hey! - this is the real world, right? Things aren’t perfect.

OK, let’s just delete references to SET_RecordIds_1020 and move on.

The next issue is downright annoying. It says Replace 'some_table' with the actual source table. Huh? It’s telling me to fill in its blanks!

I’ve given it all the information it needs. Why can’t it finish the job?

Slope of Enlightenment

Instead, I need to decipher about 200 lines of code it’s written and try to make it work since we’re nowhere near finished. And I’m not filled with confidence that it can see this task to the bitter end. At this point, I think I’m better off doing the whole thing myself.

And this is the other problem with code generated by a machine: somebody needs to maintain that stuff.

Plateau of Productivity

ChatGPT is great for simple snippets - a better StackOverflow, perhaps. But is it going to revolutionise computer coding any time soon? Well, there is an adage in software engineering that states:

The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time.

These initial results are promising but engineers won’t be out of a job any time soon.