Chapter 295 - 227: The Gap is Too Big

Chapter 295: Chapter 227: The Gap is Too Big

「December 17th, Wednesday, 9:00 AM.」

The atmosphere in the Yanhai Company office was a little different.

Usually at this time, everyone would be working on their own tasks, occasionally discussing a few technical issues.

But today was different.

Everyone was gathered around Bai Qian’s workstation, staring at the laptop in front of him, which was connected to a cloud server.

On the screen was a monitoring panel, displaying the real-time status of 16 A100 GPUs.

Temperature, power consumption, VRAM usage, compute utilization...

The numbers jumped and flickered, just like the machines in a hospital that monitor a patient’s vital signs.

"It’s now nine twenty-eight."

Bai Qian glanced at his watch, took a deep breath, and hovered his fingers over the keyboard.

"Dataset path configured, training script checked, distributed communication test passed."

He turned his head to look at Chen Lin, who was standing behind him.

"President Chen, all preparations are complete. Shall I begin?"

Chen Lin nodded, his tone calm.

"Begin."

Bai Qian firmly pressed the Enter key.

On the screen, the black terminal window instantly began to scroll wildly with dense log information.

[INFO] Distributed training initialized. World size: 16

[INFO] Loading tokenized dataset...

[INFO] Dataset loaded. Total tokens: 1.02B

[INFO] Model architecture: AdaptiveHybridTransformer-1.5B

[INFO] Training configuration: 20B tokens, batch size 2048, learning rate 1e-4

[INFO] Starting training...

[Step 1/9765625] Loss: 8.2341, LR: 1.0000e-05, Tokens/sec: 524288

[Step 2/9765625] Loss: 8.1987, LR: 2.0000e-05, Tokens/sec: 531456

Chen Lin stood in the middle of the crowd, watching the numbers flicker on the screen, feeling quite nervous inside.

Although the tests a few days ago had already proven that the new architecture’s code could run, that was just the most basic forward propagation test.

The real training process was the true litmus test for this architecture.

Just then, Bai Qian suddenly frowned.

"Hmm?"

He stared at the loss curve on the screen, his expression growing serious.

Standing beside him, Pan Si noticed it too. He pushed up his glasses and leaned in closer.

"This loss reduction rate... seems a bit slow."

Bai Qian nodded.

"It is. Logically, the initial stage of training should be when the loss drops the fastest. But look at this curve, it’s almost flat."

Hearing this, Chen Lin’s heart sank.

Although he had spent six months teaching himself about AI, it was all theoretical. He had absolutely no experience with the various pitfalls that could occur during the training process.

"Dr. Bai, what’s the problem?" Chen Lin asked.

Bai Qian didn’t answer immediately. Instead, he quickly tapped on the keyboard, pulling up a more detailed training log.

He stared at the screen for a long while before speaking slowly.

"President Chen, please come take a look."

Chen Lin walked over to Bai Qian’s side and followed his finger to the screen.

"Look here." Bai Qian pointed to a line of data. "The initial learning rate is 1e-4, which is 0.0001. This value... is a bit conservative."

"Conservative?" Chen Lin was puzzled.

"Right." Bai Qian nodded. "The learning rate is like a car’s gas pedal. If you don’t press it hard enough, the car won’t go fast."

"In the early stage of training, the model’s parameters are all randomly initialized, so they’re far from the optimal solution. This is when we should use a larger learning rate to let the model quickly find a general direction."

As he spoke, Bai Qian brought up another chart.

"You see, the current gradient norm is very stable, which means the training process is safe and we won’t encounter gradient explosion."

"Under these circumstances, we can absolutely increase the learning rate to 5e-4, which is 0.0005."

"And..." Bai Qian paused. "I suggest adding a warm-up strategy."

"Warm-up?" Chen Lin repeated the word.

"Yes." Bai Qian explained, "Warm-up means ’to get warmed up.’ The specific method is to start the training with a very small learning rate, say 1e-7, and then linearly increase it to the target value of 5e-4."

"This has two benefits."

Bai Qian held up two fingers.

"First, it prevents the model from crashing right at the start due to large gradients caused by poor parameter initialization."

"Second, it gives the model an ’adaptation’ period at the beginning to find a relatively stable optimization path, making subsequent training smoother."

After listening, Chen Lin nodded thoughtfully.

Although he didn’t understand the specific details of training, Bai Qian’s explanation was clear and his logic was sound.

More importantly, Chen Lin trusted Bai Qian’s professional expertise.

"Then do as you say," Chen Lin said decisively. "Adjust the learning rate to 5e-4 and add the warm-up."

"Got it!"

Bai Qian immediately began modifying the training script.

His fingers flew across the keyboard, and a few minutes later, the new configuration file was ready.

"President Chen, I’m restarting the training now, is that okay?"

"Go ahead."

Bai Qian pressed Enter. The training process was terminated and then immediately restarted with the new configuration.

This time, the loss curve on the screen immediately showed a difference.

[Step 1/9765625] Loss: 8.2341, LR: 1.0000e-07, Tokens/sec: 524288

[Step 100/9765625] Loss: 7.8234, LR: 5.0000e-06, Tokens/sec: 528192

[Step 500/9765625] Loss: 6.4521, LR: 5.0000e-05, Tokens/sec: 534528

[Step 1000/9765625] Loss: 5.2347, LR: 2.5000e-04, Tokens/sec: 537856

"Look!" Bai Qian pointed excitedly at the screen. "The loss is decreasing much faster now!"

Pan Si also nodded.

"Yeah, this curve looks much more normal."

Watching the steadily descending curve on the screen, Chen Lin felt the weight on his heart finally lift.

He patted Bai Qian’s shoulder.

"Nice work."

Bai Qian chuckled.

"This is all standard procedure. Training a large model is just a constant process of tuning parameters, observing, and tuning again."

"Now, we just keep an eye on this curve and see what surprises it has for us."

...

Over the next two days, a peculiar "sight" became a new fixture in the Yanhai Company office.

Bai Qian’s workstation became the center of everyone’s attention.

The laptop connected to the cloud server ran 24/7, its monitoring panel becoming the thing everyone cared about most.

The first thing Sun Yu did every day when he got to the office was to lean in front of the screen to see how far the loss had dropped.

Although Shen Yan appeared calm on the surface, her eyes would involuntarily dart toward that screen every time she passed Bai Qian’s workstation.

Even Yunqing, the "layperson," was starting to understand the loss curve.

"Hey, the line is steeper today than yesterday!"

"Right? See! I told you our new architecture is amazing!"

「December 18th, Thursday, 3:00 PM.」

The training had already processed 5 billion tokens.

Bai Qian, staring at the screen, suddenly had his eyes light up.

"President Chen!"

He spun around, his voice filled with unconcealed excitement.

"We can run a mid-training evaluation!"

Chen Lin, who had been discussing data cleaning with Pan Si, immediately walked over when he heard Bai Qian’s shout.

"What kind of evaluation?"

"A text generation test," Bai Qian said. "Although the model isn’t finished training, we can already use the current checkpoint for inference."

"I want to see what kind of text it can generate now."

"Then let’s try it," Chen Lin nodded.

Bai Qian immediately got to work.

He downloaded the current model weight file from the cloud server and then launched an inference script locally.

"Give me a prompt," Bai Qian said.

Chen Lin thought for a moment, then said casually:

"Just write... ’The future development direction of artificial intelligence is’."

Bai Qian typed the sentence into the script and pressed Enter.

A few seconds later, the model’s generated text began to appear on the screen, word by word:

The future development direction of artificial intelligence is the fusion of multi-modalities and the enhancement of generalization ability.

Current AI systems mostly focus on a single task, such as natural language processing, computer vision, or speech recognition.

However, true Artificial General Intelligence (AGI) requires the ability to understand and reason across modalities.

Future research will focus on how to enable models to simultaneously process multiple data forms like text, images, and audio, and to achieve knowledge transfer between different tasks.

Furthermore, how to train more efficient and robust models under conditions of limited data and computing power is also a significant research direction...

Watching the smooth, coherent, and logically clear text on the screen, a silence fell over the office.

Everyone’s eyes were wide.

"Holy shit..."

Even Sun Yu, who wasn’t an AI expert, could tell this was something special.

"This... this was generated by a small 1.5B model?"

Bai Qian’s hands were trembling slightly.

He immediately entered another prompt for a comparison test.

"I’m going to use the same prompt and have the baseline model—Llama-1.5B—generate a piece of text. You guys can see the difference."

A few seconds later, the output from Llama-1.5B appeared:

The future development direction of artificial intelligence is very good, I think AI will become more and more amazing, it can do many things, like helping people with work, and life, etc.

Future AI should be more intelligent, able to understand human needs, and provide better services. In short, the development prospects for AI are very broad...

Although this text was also coherent, it clearly lacked depth, felt like it was just padding the word count, and had obvious repetition and verbosity.

In comparison, the text generated by the new model was a strike from a higher dimension.

"The difference..." Pan Si pushed up his glasses and murmured, "is just too obvious."