It may be worth abandoning hopes of a career in alert engineering, as new research from VMware Labs shows that AI models are much better suited to generating their own alerts.
Based on tests that compared human-generated “positive thinking” prompts with “systematic prompt optimization” or AI-generated prompts, researchers found that automated processes yielded better results.
Lead researchers Rick Battle and Teja Gollapudi found that using an “automatic prompt optimizer” was the most effective method for “improving performance” regardless of model size.
The study also found that the AI-optimized prompts that scored highest in their testing metrics also exhibited an unexpected “degree of quirk.”
“As anticipated, prompts that underwent automatic optimization consistently matched or exceeded the effectiveness of our manually generated 'positive thinking' prompts in almost all cases,” the research said.
Although the paper describes the act of rapid engineering as an “enjoyable endeavor,” the researchers also argued that the study clearly shows that human rapid engineering is “time-inefficient.”
This was especially true, the researchers said, when systematically evaluating all the modifications from a scientific point of view. Cases where human cue engineering outperformed automated cueing were limited to a handful of tests on Mistral-7B and Llama2-70B.
The research indicated almost no clear trends when approaching rapid engineering from a human-generated approach: “the only real trend may be the absence of a trend,” he said.
In light of the data, the two researchers laid out their rationale for rapid engineering as a whole.
“What is best for any given model, data set, and stimulus strategy is likely specific to the particular combination in question. “Therefore, we move from manually adjusting the system message with optimistic “positive thinking” to rapid and automatic optimization,” the research says.
Rapid engineering is not a safe bet in the job market
Rapid engineering is a hugely important process in AI development, as fine-tuning AI inputs helps models operate to their fullest extent, using potentially untapped aspects of their data, according to Bartek Roszak, Head of AI at STX Next.
“Rapid engineering is important because it allows us to extract precisely what we need from LLMs,” he said. ITPro.
However, as this new research suggests, this is a task increasingly vulnerable to automation given the exact technology you seek to use.
Expanding your research, Battle concluded that humans should never again have to manually optimize a message. Battle further criticized the process of a human engineer wasting time having to “figure out” an effective combination of words.
Battle hopes this research will convince future prompt engineers to move beyond manual prompt engineering, advising AI users to instead “develop a scoring metric” so the model can train itself.
Roszak also looked toward a future of AI autonomy in the realm of rapid engineering, suggesting that this will eventually become an obsolete skill.
“Ideally, everyone would be able to interact with the LLMs, and if someone provided a vague indication, the LLM would participate by asking for details and help refine the indication itself,” Roszak said.
“These advances could lead to a future where rapid and deep engineering is no longer a specialized skill,” he added.
If AIs can be trained to create their own ads, the role of the human ads engineer could be dead.
However, that doesn't mean there aren't other areas where human involvement abounds in the realm of AI optimization.
“The evolution of career paths in AI will likely shift toward oversight, ethical programming, and innovative application of AI results,” said Peter Wood, CTO of Spectrum Search. ITPro.
According to Peter Van Der Putten, chief scientist and director of Pegasystems, a different type of fast engineer could also remain. Speaking to ITPro, Van Der Putten said that in the long term there will likely be a greater focus on a more global, enterprise-level understanding of rapid engineering.
“Prompt engineers will focus much more on industrialization, from creating dynamic Prompt templates, connecting AI with knowledge bases and search engines, to building advanced agent infrastructures,” said Van Der Putten .
While rapid engineering remains a necessity in the burgeoning field of training and using AI, tech industry staff will need to ensure they are ahead of the curve if they want their skills to be useful.
“At this time, the work of a punctual engineer is key to the effective use of LLMs; however, research is actively seeking ways to make this work obsolete,” Roszak said.
