Chapter 296 - 228: Concentrating Strength

Chapter 296: Chapter 228: Concentrating Strength

Bai Qian took a deep breath and continued,

"And it gets even better."

He switched to another test script.

"I’m going to test its long-text generation capabilities now."

Bai Qian entered a longer prompt, about 500 tokens, and then had the model generate another 2,000 tokens of content.

The new model’s output was still smooth, coherent, and logically sound, showing almost no signs of "attention collapse."

But when the same task was given to Llama-1.5B...

"Look here." Bai Qian pointed at the screen. "Around the 1,500-token mark, it starts spewing nonsense."

On the screen, the text generated by Llama-1.5B suddenly became incoherent from a certain point onward. It was filled with repetitive phrases and even some meaningless gibberish.

This was a classic case of "attention collapse."

When the input sequence is too long, the attention mechanism of the traditional Transformer architecture fails. This causes the model to "forget" the preceding content and get stuck in a loop.

But the new architecture...

...had no such problem at all!

Looking at the results of these comparison tests, Chen Lin was absolutely ecstatic.

He couldn’t help but blurt out,

"Holy shit! I didn’t think this architecture would be so fucking awesome!"

After saying that, he turned his head and looked at Bai Qian excitedly.

"Dr. Bai, once the 30B model is trained, won’t it be able to wipe the floor with all the open-source models on the market?"

Hearing this, Bai Qian didn’t share his excitement.

He pushed up his glasses and said calmly,

"President Chen, let’s not get ahead of ourselves."

"We’re only at 5B tokens. It’s still too early."

"We need to wait until at least the 15B token mark to get stable benchmark data before we can draw any conclusions."

Chen Lin was taken aback for a moment, then he smiled.

"Alright, your call. We’ll keep observing."

Although he said that, Chen Lin couldn’t suppress the excitement bubbling up inside him.

He knew exactly what this meant.

If this new architecture was really as powerful as the test results showed...

...then Yanhai Company would have an absolute, crushing technological advantage in the race for large AI models!

...

「December 19th, Friday, 10:00 AM.」

Training progress: 10B tokens.

Bai Qian and Chen Lin were sitting in front of a computer, reviewing the latest evaluation report.

This report was generated by the automated test script Bai Qian had set up the previous night.

It contained several industry-standard benchmarks:

MMLU (Massive Multitask Language Understanding): Tests the model’s performance on knowledge across various academic subjects.

GSM8K: A test of grade-school math word problems, examining the model’s mathematical reasoning abilities.

HumanEval: A code generation test that provides a function description and requires the model to generate executable code.

Bai Qian opened the report, and a detailed comparison table appeared on the screen:

text

| Benchmark | Yanhai-1.5B (10B tokens)| Llama-1.5B | Llama-7B |

|-----------|------------------------|------------|----------|

| MMLU | 58.3%| 42.1%| 62.5%|

| GSM8K | 34.7%| 11.2%| 47.3%|

| HumanEval | 23.5%| 8.1%| 29.8%|

Seeing this table, Chen Lin was completely stunned.

"This..."

He rubbed his eyes, suspecting he was seeing things.

"Dr. Bai, are these numbers... real?"

Bai Qian nodded, a satisfied smile on his face.

"Absolutely. I ran it three times last night, and the results were more or less the same."

Chen Lin took a deep breath, trying to calm himself down.

But he couldn’t hide the shock he felt.

An MMLU score of 58.3!

Keep in mind, Llama-1.5B only scored 42.1!

And Llama-7B was only at 62.5!

What did this mean?

It meant that the 1.5B-parameter model trained by Yanhai Company was already approaching the performance of the 7B-parameter Llama in multitask language understanding!

With only one-fifth of the parameters, its performance was on par!

This was practically magic!

Chen Lin’s gaze moved further down the table.

GSM8K, an accuracy of 34.7%!

Llama-1.5B only had 11.2%!

That was a three-fold difference!

HumanEval, a pass rate of 23.5%!

Llama-1.5B only had 8.1%!

That was almost a three-fold difference!

Chen Lin leaned back in his chair, his mind a complete blank.

He never would have imagined that the new architecture he had deduced could be this powerful.

Just then, Pan Si, who was sitting to the side, suddenly spoke up.

He pushed up his glasses, his tone laced with a complex mix of emotions.

"Dr. Bai, congratulations."

"The potential of this new architecture has indeed exceeded my expectations."

Bai Qian turned to look at Pan Si and smiled.

"Dr. Pan, you sound a little salty, don’t you think?"

Pan Si gave a wry smile.

"I am a little salty."

He paused, then continued,

"My 30B model has been trained on 100B tokens already."

"And while its performance has reached the industry average, compared to your 1.5B little monster..."

Pan Si shook his head, not finishing his sentence.

Sun Yu, who was standing nearby, looked disheartened upon hearing this.

"Dr. Pan, does that mean all our work during this time... was for nothing?"

"It wasn’t for nothing."

Bai Qian immediately interjected.

He looked at Sun Yu and said earnestly,

"Sun, you can’t think of it that way."

"For Dr. Pan’s 30B model to reach the industry average is already quite an achievement."

"You have to understand, the big tech companies use far more computing resources than we do when they train their models."

"We only have 16 A100s, while they casually use clusters of thousands of GPUs."

"Besides, training a large model isn’t just about computing power for training; you also need resources for inference."

"The evaluation tests for Dr. Pan’s project were also run on our 16 A100s. A portion of these GPUs has to be used for inference while the rest continue training, so resources are already tight."

"So, it’s very likely that the 30B model’s true performance hasn’t been fully realized yet."

"To achieve the industry average under these conditions is a testament to how effective Dr. Pan’s data cleaning and alignment strategies are."

Hearing Bai Qian’s words, the color returned to Sun Yu’s face.

Pan Si also nodded.

"Thank you, Dr. Bai."

He paused, his gaze becoming resolute.

"However, I must also admit that my previous opposition to the new algorithm architecture was indeed a bit arbitrary."

"I thought that improving the existing architecture and enhancing performance through data and engineering optimization was the safest route."

"But I overlooked one thing..."

Pan Si looked at Chen Lin, his tone filled with admiration.

"President Chen’s mathematical prowess can play a huge role in deducing a new framework."

"That’s something I hadn’t considered before."

After speaking, Pan Si took a deep breath, stood up, and walked over to Chen Lin.

"President Chen, I have a suggestion."

"Go on," Chen Lin said, looking at him.

Pan Si’s tone was incredibly solemn.

"Should we consider pausing my project?"

"I’d like to take Sun Yu and Shen Yan and transfer over to Dr. Bai’s team to help him accelerate the training process for the 1.5B model."

"And then, we can start training the 30B model as soon as possible."

Chen Lin was taken aback.

He hadn’t expected Pan Si to propose this himself.

You have to understand, that 30B model project was Pan Si’s own "baby."

To give it up voluntarily now... what incredible decisiveness and magnanimity did that require?

Chen Lin was silent for a moment.

He looked at Pan Si, then at Bai Qian, and finally his gaze swept over Sun Yu and Shen Yan.

"Dr. Pan, are you sure?"

"I’m sure." Pan Si nodded. "Under our limited resources, concentrating our efforts to achieve a major goal is the optimal strategy."

"Besides..."

Pan Si looked at Bai Qian, a slight smile playing on his lips.

"I also want to personally participate in the training process of this new architecture and see just how powerful it can truly be."

Hearing this, Bai Qian’s eyes lit up.

"Dr. Pan, if you’re willing to come help, I couldn’t ask for more!"

"I was just worrying about not having enough people!"

Watching the two of them and their tacit understanding, Chen Lin felt a strong sense of being moved.

’This is what a team is.’

Even though they had disagreements on the technical approach, when faced with a common goal, everyone could put aside their prejudices and work together.

"Good."

Chen Lin took a deep breath and made his decision.

"Dr. Pan, your 30B model project is paused."

"You, Sun Yu, and Shen Yan will all transfer to Dr. Bai’s team to assist him in completing the training and evaluation of the 1.5B model."

"And then..."

Chen Lin’s gaze became incredibly determined.

"We go all out and begin training the 30B model!"