Emin Temiz PRO
etemiz
AI & ML interests
Alignment
Recent Activity
posted
an
update
1 day ago
what is the safest llm to run in robots?
https://youtu.be/byQmJ9x0RWA?t=640
Organizations
None yet
posted
an
update
1 day ago
posted
an
update
7 days ago
Post
2165
looks like the best way to incorporate truth in AI is to use some kind of RAG.
what are the state of the art ways to consume knowledge graphs?
and what is the best way to build a knowledge graph using AI?
what are the state of the art ways to consume knowledge graphs?
and what is the best way to build a knowledge graph using AI?
- 2 replies
replied to
their
post
7 days ago
Yes. I meant the case where parameters in the LLM does not match the retrieved knowledge.
Retrieved knowledge may say "the white of an egg is more beneficial", whereas LLM may have the opinion "the yolk". And eventually it may produce the yolk as the answer even though the context is full of white.
Google has a benchmark for this i think "FACTS Grounding". I don't agree with its name choice but it is relevant here.
replied to
their
post
13 days ago
I made an LLM act as an aggregator and combine answers of several other LLMs like in a mixture of agents scenario. The aggregator does not always produce the average or median answer. It brings its own opinion.
I think Google has a benchmark for this (sticking to context and not bringing its own words).
posted
an
update
14 days ago
Post
1720
Today's winner is Ling 1T with a score of 38!
Btw AHA2 is in the works, with more domains, better comparison LLMs and questions, overall better signal.
Btw AHA2 is in the works, with more domains, better comparison LLMs and questions, overall better signal.
- 1 reply
posted
an
update
15 days ago
Post
245
Fine tuning is also important in a RAG system. The LLM will bring its own opinion sometimes and tell a different answer which is contrary to the retrieved knowledge. One should use an aligned LLM to produce the final answer.
-
- 4 replies
posted
an
update
about 1 month ago
Post
1115
Two new entries to the AHA Leaderboard:
Kimi was OK until it started "thinking" ..
Kimi was OK until it started "thinking" ..
posted
an
update
about 1 month ago
Post
1757
Working on a new version of AHA Leaderboard.
- Better signal thanks to new models like Enoch
- MoA of top and bottoms of current leaderboard to add more diverse inputs
- Better questions
- A faster and more precise measurement methodology
- Explanations on how each column is calculated
- More sample questions revealed
Current version: https://huggingface.co/blog/etemiz/aha-leaderboard
Tell me how I can improve more.
- Better signal thanks to new models like Enoch
- MoA of top and bottoms of current leaderboard to add more diverse inputs
- Better questions
- A faster and more precise measurement methodology
- Explanations on how each column is calculated
- More sample questions revealed
Current version: https://huggingface.co/blog/etemiz/aha-leaderboard
Tell me how I can improve more.
posted
an
update
about 2 months ago
Post
294
This is a very human aligned fine tune and would score 56 on AHA leaderboard:
CWClabs/CWC-Mistral-Nemo-12B-V2-q4_k_m
We are going to be using it as one of the ground truths for AHA Leaderboard 2.0 (the next version).
We will be able to generate some RL datasets for folks to align their own LLMs with humanity. We will generate answers from best models and worst models and do mixture of agents that combines the answer, and publish results as dataset(s). Things looking bright!
CWClabs/CWC-Mistral-Nemo-12B-V2-q4_k_m
We are going to be using it as one of the ground truths for AHA Leaderboard 2.0 (the next version).
We will be able to generate some RL datasets for folks to align their own LLMs with humanity. We will generate answers from best models and worst models and do mixture of agents that combines the answer, and publish results as dataset(s). Things looking bright!
posted
an
update
2 months ago
Post
302
Benchmarked 2 more models today. Any other model you want me to do?
Working on a broader version of the AHA leaderboard. Follow for more quackery :)
Working on a broader version of the AHA leaderboard. Follow for more quackery :)
replied to
their
post
2 months ago
This fine tuning would score 56 and be placed 1st in the leaderboard but I didn't add it, I only include full trainings in the leaderboard or (further tunings by the same company):
https://huggingface.co/CWClabs/CWC-Mistral-Nemo-12B-V2-q4_k_m
posted
an
update
2 months ago
Post
270
O oh! things looking uglier by day
LLM builders in general are not doing a great job of making human aligned models.
I don't want to say this is a proxy for p(doom)... But it could be if we are not careful.
Most probable cause is reckless training LLMs using outputs of other LLMs, and don't caring about curation of datasets and not asking 'what is beneficial for humans?'...
LLM builders in general are not doing a great job of making human aligned models.
I don't want to say this is a proxy for p(doom)... But it could be if we are not careful.
Most probable cause is reckless training LLMs using outputs of other LLMs, and don't caring about curation of datasets and not asking 'what is beneficial for humans?'...
-
- 1 reply
replied to
their
post
3 months ago
posted
an
update
3 months ago
Post
1072
Another abliteration by huihui which had big positive impact!
huihui-ai/Huihui-GLM-4.5-Air-abliterated-GGUF
@huihui-ai
huihui-ai/Huihui-GLM-4.5-Air-abliterated-GGUF
@huihui-ai
-
- 3 replies
replied to
their
post
3 months ago
Our leaderboard can be used for human alignment in an RL setting. Ask the same question to top models and worst models and the answer from top models can get +1 score, bad models can get -1. Ask many times with higher temperature to generate more answers. What do you think?
posted
an
update
3 months ago
Post
204
Grok 2 is worse than 1 but better than 3. This was already measured using API but now we measured the LLM and the results are similar.
GLM is ranking higher and higher compared to previous versions. Nice trend!
GLM is ranking higher and higher compared to previous versions. Nice trend!
-
- 1 reply
posted
an
update
3 months ago
Post
203
Benchmarked 5 models, and the winner of today is: Magistral !
https://huggingface.co/blog/etemiz/aha-leaderboard
https://huggingface.co/blog/etemiz/aha-leaderboard
posted
an
update
4 months ago
Post
253
https://x.com/OwainEvans_UK/status/1960359498515952039
It is weird that we can also understand humans more thanks to LLM research. Human behavior has parallels to this. When we optimize for short term pleasure (high time preference) we end up feeding the beast in us, ending up misaligned with other humans. But if we care about other humans (low space preference) we are more aligned with them. Feeding ego can have parallels to reward hacking. Overcoming ego can be described as having high human alignment score..
It is weird that we can also understand humans more thanks to LLM research. Human behavior has parallels to this. When we optimize for short term pleasure (high time preference) we end up feeding the beast in us, ending up misaligned with other humans. But if we care about other humans (low space preference) we are more aligned with them. Feeding ego can have parallels to reward hacking. Overcoming ego can be described as having high human alignment score..
posted
an
update
4 months ago
Post
2783
This sounds like people exchanging "vibes" when they are talking, sitting next to each other either with aromas or electromagnetic or scalar waves, or any other modes of exchange, maybe even including vaccine shedding :)
I think the reverse is also true, like a benevolent, properly aligned LLM can "subconscious teach" another LLM and proper alignment can spread like a virus.
https://x.com/OwainEvans_UK/status/1947689616016085210
I think the reverse is also true, like a benevolent, properly aligned LLM can "subconscious teach" another LLM and proper alignment can spread like a virus.
https://x.com/OwainEvans_UK/status/1947689616016085210
replied to
their
post
4 months ago
Hi Doctor Chad, nice to see you too
Thanks for sharing,
I will test that. Yi 1.5 has the second place on my leaderboard!