發(fā)布人：Google Research 研究員 Aurko Roy

開(kāi)放域長(zhǎng)格式問(wèn)答 (LFQA) 是自然語(yǔ)言處理 (NLP) 的一項(xiàng)基礎(chǔ)挑戰(zhàn)，涉及檢索與給定問(wèn)題相關(guān)的文檔，并使用這些文檔來(lái)生成一段詳盡答案。在事實(shí)型開(kāi)放域問(wèn)答 (QA) 中，簡(jiǎn)單的短語(yǔ)或?qū)嶓w便足以回答問(wèn)題。雖然我們近期在這一方面取得了顯著進(jìn)展，但在長(zhǎng)格式問(wèn)答領(lǐng)域中卻做得遠(yuǎn)遠(yuǎn)不夠。盡管如此，LFQA 仍是一項(xiàng)非常重要的任務(wù)，特別是它能提供一個(gè)測(cè)試平臺(tái)來(lái)衡量生成文本模型的真實(shí)性。但是，當(dāng)前的基準(zhǔn)和評(píng)估指標(biāo)真的能在 LFQA 方面取得進(jìn)展嗎？

在“在長(zhǎng)格式問(wèn)答領(lǐng)域取得進(jìn)展的障礙”(Hurdles to Progress in Long-form Question Answering)（將在 NAACL 2021 會(huì)議上發(fā)表）中，我們介紹了一種新的開(kāi)放域長(zhǎng)格式問(wèn)答系統(tǒng)，它利用了 NLP 的兩項(xiàng)最新進(jìn)展：

1.最先進(jìn)的稀疏注意力模型（例如 Routing Transformer(RT)），能夠?qū)⒒谧⒁饬Φ哪Ｐ蛿U(kuò)展至長(zhǎng)序列；

2.基于檢索的模型（例如 REALM），有助于檢索與給定查詢(xún)相關(guān)的維基百科文章。

Routing Transformer

https://www.mitpressjournals.org/doi/full/10.1162/tacl_a_00353

為獲得更多的事實(shí)依據(jù)，對(duì)于檢索到的與給定問(wèn)題相關(guān)的一些維基百科文章，我們的系統(tǒng)會(huì)在答案生成之前將從中獲得的信息結(jié)合起來(lái) ELI5 是唯一一個(gè)可用于長(zhǎng)格式問(wèn)答的大規(guī)模公開(kāi)數(shù)據(jù)集，我們的系統(tǒng)在該數(shù)據(jù)集上取得了突破性進(jìn)展。

ELI5

https://ai.facebook.com/blog/longform-qa/

不過(guò)，雖然這個(gè)系統(tǒng)在公開(kāi)排行榜上名列前茅，但我們發(fā)現(xiàn) ELI5 數(shù)據(jù)集及其相關(guān)評(píng)估指標(biāo)的一些趨勢(shì)令人擔(dān)憂(yōu)。特別要強(qiáng)調(diào)的是，我們發(fā)現(xiàn) 1) 幾乎沒(méi)有證據(jù)表明模型實(shí)際使用了它們所要求的檢索；2) 平凡基線（例如輸入復(fù)制）擊敗了現(xiàn)代系統(tǒng)，如 RAG/BART+DPR；以及 3) 數(shù)據(jù)集中存在大量訓(xùn)練/驗(yàn)證重疊。我們的論文針對(duì)每一個(gè)問(wèn)題提出了緩解策略。

輸入復(fù)制

https://eval.ai/web/challenges/challenge-page/689/leaderboard/1908#leaderboardrank-6

文本生成

NLP 模型的核心要件是 Transformer 架構(gòu)，其序列中的每個(gè) Token 都會(huì)關(guān)注序列中的其他所有 Toekn，從而形成一個(gè)隨序列長(zhǎng)度呈二次增長(zhǎng)的模型。RT 模型引入了一種基于內(nèi)容的動(dòng)態(tài)稀疏注意力機(jī)制，將 Transformer 模型中的注意力復(fù)雜度從 n2 降到了 n1.5（ 其中 n 是序列長(zhǎng)度），使其能夠擴(kuò)展到長(zhǎng)序列。這使得每個(gè)單詞都可以關(guān)注整個(gè)文本中 任何地方的其他相關(guān)單詞， 而不像 Transformer XL 等類(lèi)似方法，一個(gè)單詞只能關(guān)注其附近的單詞。

RT 發(fā)揮作用的關(guān)鍵在于每個(gè) Token 對(duì)其他 Token 的關(guān)注通常是冗余的，并且可以通過(guò)結(jié)合局部和全局注意力進(jìn)行估算。局部注意力允許每個(gè) Token 在模型的幾個(gè)層上建立一個(gè)局部表征，其中每個(gè) Token 關(guān)注一個(gè)局部鄰域，從而達(dá)到局部的一致性和流暢性。作為對(duì)局部注意力的補(bǔ)充，RT 模型還使用了小批量 k-均值集群， 使每個(gè) Token 只關(guān)注一組最相關(guān)的 Token 。

我們以語(yǔ)言建模為目標(biāo)，使用 ProjectGutenberg(PG-19) 數(shù)據(jù)集預(yù)先訓(xùn)練了一個(gè) RT 模型，即在給定前面所有單詞的情況下，讓該模型學(xué)會(huì)預(yù)測(cè)下一個(gè)單詞，從而能夠生成流利的段落長(zhǎng)文本。

ProjectGutenberg(PG-19)

https://deepmind.com/blog/article/A_new_model_and_dataset_for_long-range_memory

信息檢索

為了證明 RT 模型在 LFQA 任務(wù)中的有效性，我們將其與 REALM 中檢索到的內(nèi)容結(jié)合使用。REALM 模型（Guu 等人于 2020 年發(fā)布）是基于檢索的模型，使用最大內(nèi)積搜索來(lái)檢索與特定查詢(xún)或問(wèn)題相關(guān)的維基百科文章。我們對(duì)該模型進(jìn)行了微調(diào)，以便根據(jù)自然問(wèn)題數(shù)據(jù)集作出事實(shí)型問(wèn)答。REALM 利用 BERT 模型學(xué)習(xí)問(wèn)題的良好表征，并使用 SCANN 檢索與問(wèn)題表征具有高度主題相似性的維基百科文章。接著進(jìn)行端到端訓(xùn)練，以最大程度地提高 QA 任務(wù)的對(duì)數(shù)似然值。

通過(guò)使用對(duì)比損失，我們進(jìn)一步提高了 REALM 檢索的質(zhì)量。其背后的想法是讓問(wèn)題表征更靠近其基本事實(shí)答案，并與其他小批量答案有所不同。這樣可以確保，當(dāng)系統(tǒng)使用此問(wèn)題表征檢索相關(guān)項(xiàng)目時(shí)，會(huì)返回與基本事實(shí)答案“類(lèi)似”的文章。我們稱(chēng)這種檢索器為對(duì)比型-REALM 或 c-REALM。

對(duì)比損失

https://towardsdatascience.com/contrastive-loss-explaned-159f2d4a87ec

評(píng)估

我們使用 ELI5 數(shù)據(jù)集對(duì)該模型進(jìn)行了長(zhǎng)格式問(wèn)答測(cè)試。ELI5 數(shù)據(jù)集是 KILT 基準(zhǔn)的一部分， 也是唯一 公開(kāi)可用的大規(guī)模 LFQA 數(shù)據(jù)集。KILT 基準(zhǔn)使用準(zhǔn)確率 (R-Prec) 衡量文本檢索質(zhì)量，并使用 ROUGE-L 衡量文本生成質(zhì)量。將這兩項(xiàng)得分相加即可得出一個(gè) KILT R-L 得分，該得分決定了模型在排行榜上的排名。在 KILT 的 ELI5 數(shù)據(jù)集上，我們對(duì)預(yù)先訓(xùn)練好的 RT 模型和 c-REALM 檢索進(jìn)行了微調(diào)。

我們提交的內(nèi)容在 ELI5 長(zhǎng)格式問(wèn)答 KILT 排行榜上名列前茅，綜合 KILT R-L 得分為 2.36。對(duì)比先前 BART + DPR 排行榜的排名（KILT R-L 得分為 1.9）有了提升，同時(shí)具有與排行榜上其他模型相似的參數(shù)數(shù)量。在文本生成質(zhì)量方面，我們發(fā)現(xiàn) Rouge-L 與 T5、BART + DPR 和 RAG 相比，得分分別提高了 4.11、5.78 和 9.14。

排行榜

https://eval.ai/web/challenges/challenge-page/689/leaderboard/1908

RT 模型生成的示例

Q1

Why are almost all boats white?

Boats are generally painted white for obvious reasons, as they are inexpensive and easy to clean. Ships are also often painted to provide a visible visual contrast to the sea. Boats also often have high visibility ports to allow for easy international trade.

In addition, many boats have an air-breathing function that makes them easier to spot and stay afloat, which is a good thing if you're a pirate or trying to get into some country's waters. Lastly, many ships are built in the last few decades, so they are not as exotic looking as they were thirty years ago.

And lastly, many ships are made out of materials that are easy to work with, and so many of them are painted white as a practical matter. There are many, many more reasons for why ships are white, but those are the ones I know of. I'm sure there are many other reasons, but those are the ones I know of.

Edit: I forgot to mention that the other answers about visibility and easy to see ports are also true. Ships are painted to appear to be larger, and more expensive. That makes them appear more impressive, and so they are more likely to be bought by people who want to show off their wealth and status.

Q2

What causes the trail behind jets at high altitude?

The same thing that causes the clouds you see at the start of a storm to form in the first place. The air pressure changes as the jet moves through the air, causing it to cool down and condense into water droplets. These droplets are in fact a lot bigger than the water droplets in the clouds, so when the plane passes through them, they can cause the air pressure to change, causing the water droplets to condense into clouds.

This is what you see when you look at a cloud formation from the ground. In addition to this, when the plane passes through the clouds, it also moves through the air, causing it to change its temperature, causing the water droplets to condense and form clouds.

These clouds are often quite large, and can block out the sun. This is what you see when you look at a contrail. Finally, the water vapor that forms at high altitudes is cooled by the air above, and as it cools, it condenses into clouds. This is what you see when you look at a cumulonimbus cloud. These clouds are even larger than the water droplets in the clouds, and they block out the sun.

Q3

How do shampoo and conditioner work?

The key ingredients in shampoo and conditioner are usually surfactants, they tend to attract oil from your skin. Shampoo will usually contain an anti-dandruff agent (dandruff is caused by fungi in your skin which cause dandruff). They also contain some moisturizing agents (moisturizers are things that stay in your skin longer and protect your scalp from cleaning),

And a lot of anti-dandruff (dandruff is a side effect of high doses of both surfactants and moisturizers) Conditioner will contain moisturizing agents, like a skin moisturizer. They tend to cause oils to be separated from your hair (e.g. wash your hair with conditioner instead of shampoo). They both have a foaming agent that makes them foam up (usually Hydrogen Peroxide). They also have a foaming agent that stops dirt from attaching to your hair. Some conditioner will also suppress the action of the shampoo (to help prevent dirt from attaching to your hair).

Q4

When glancing at a clock, why does the first second after glancing at it sometimes feel longer than the rest?

This is known as chronostasis. Your brain gets used to seeing the second hand move after the first, so it ignores the first one. If you were to look at the clock a lot, you'd see the second hand move a lot more than the first, and you'd be able to tell the difference between the second and the first.

The second hand will appear to move much slower than the first hand because your brain is used to seeing the second hand move a lot more than the first. If you were to go on a long trip, you'd get used to the second hand moving a lot more than the first, and you'd have no problem telling the difference between the second and the first hand.

It's kind of like how you don't notice your nose when you're driving, but if you look at it for a while, you start to notice it. Your brain is just getting used to seeing the nose move a lot more than the other, and it can't tell the difference between the two. It's kind of like that with your eyes.

Q5

Why do Britain and other English empire countries still bow to monarchs? What real purpose does the queen serve?

Depends what you define as "purpose". There is no specific purpose for the British monarch - rather, they are figureheads that serve a role. The role is to have a high degree of influence over a country, though the rules of the monarchy can somewhat vary from country to country, and the ceremony of "kissing the royal corpse" does not always involve kissing the monarch. Whether that's the only reason for the custom, or if it was the only reason,

I can't say, but that's the reasoning. When the U.K. was in full power, the monarch was a puppet of Parliament, and the powers of the monarch were transferred to the Prime Minister and thus to the Prime Minister's deputy, who then became the Prime Minister. Since then, Parliament has been able to vote on legislation that goes through the monarch,

although they may still act as the monarch's representative in negotiating treaties, which can have very very deep consequences. The Queen's role, as a representative of Britain, doesn't necessarily involve her formal approval of any of the laws or legislation that goes through Parliament, though.

Q6

What exactly is fire, in detail? How can light and heat come from something we can't really touch?

Fire is the chemical reaction of fuel (oxygen) with (what we call) impurities (ash, soot, oil etc). These impurities are created due to a number of things. If you were to study the chemical composition of a log (or a candle) and analyse it you would discover the chemical formula (A, C, and G) which is a simplified version of what actually happens. There is always fuel, and there is always the ash/soot. Wood is made up of mostly carbon and hydrogen and these are the fuels that create the fire. Hydrogen and oxygen are reactive, and they form water (H2O) and CO2 (CO2). If we split a log into smaller pieces, we start to see the process of combustion as it occurs. The carbon is burned, and with the explosion of CO, H2O and H2CO2, we get water and carbon dioxide (mostly). If we smash a log together, we start to see what happens when we also hit another log with another log...imagine smashing two sand castles together. We see how the sand castles and brick castles form and collapse at different rates...then it starts to burn. The smoke, ash and flames are the actual combustion of fuel.

Q7

What goes on in those tall tower buildings owned by major banks?

The actual buildings are not there for profit but for show. There are many real world historical buildings that have little to no effect other than being a symbol of an economic entity or symbol of an era or location, and nothing more. For example look at Sears, Sears didn't care what went on inside, it was all about the \_appearance\_ of its location, the prestige of the location, the facilities and so on. It didn't care about how long it took it to operate, it was about how much people would pay to go see it. Sears was a landmark as a cultural movement and other big companies followed suit, so if you want to see a building you've never seen before, you have to go see Sears, just like you have to see a Toyota Camry for Toyota Camry. They used to be all about building new factories, some of them if I recall, but now that they're bigger, that means that more factory jobs are coming to them. You've probably seen them in stores as stores where people buy and sell stuff, so there aren't that many places for them to come from. Instead, it's just for show, a symbol of rich people.

在 LFQA 領(lǐng)域取得進(jìn)展的障礙

不過(guò)，雖然此處介紹的 RT 系統(tǒng)在公開(kāi)排行榜中名列前茅，但對(duì)模型和 ELI5 數(shù)據(jù)庫(kù)的詳細(xì)分析仍揭示了一些令人擔(dān)憂(yōu)的趨勢(shì)。

Train/Valid Overlap

Many held-out questions are paraphrased in the training set. Best answer to similar train questions gets 27.4 ROUGE-L.

Lack of Grounding

Conditioning answer generation on random documents instead of relevant ones does not measurably impact its factual correctness. Longer outputs get higher ROUGE-L.

我們發(fā)現(xiàn)，幾乎沒(méi)有任何證據(jù)表明模型會(huì)將其文本生成實(shí)際定位到檢索文檔中。與 Wikipedia 中的隨機(jī)檢索搭配使用的微調(diào) RT 模型（例如，隨機(jī)檢索 + RT），幾乎與 c-REALM + RT 模型（24.2 與 24.4 ROUGE-L）表現(xiàn)得一樣好。在訓(xùn)練、驗(yàn)證和測(cè)試 ELI5 數(shù)據(jù)集時(shí)，我們還發(fā)現(xiàn)了很多的重疊（幾個(gè)問(wèn)題相互解釋?zhuān)?，因此可能不再需要檢索。KILT 基準(zhǔn)會(huì)單獨(dú)衡量檢索和生成的質(zhì)量，但不確定文本生成是否會(huì)在實(shí)際情況中使用檢索。

與 RAG 和 BART + DPR 相比，平凡基線會(huì)獲得更高的 Rouge-L 分?jǐn)?shù)

此外，在使用 Rouge-L 指標(biāo)和平凡無(wú)意義基線（如隨機(jī)訓(xùn)練集答案和輸入復(fù)制）來(lái)評(píng)估文本生成質(zhì)量的過(guò)程中，我們發(fā)現(xiàn)了一些問(wèn)題，并導(dǎo)致 Rouge-L 分?jǐn)?shù)相對(duì)較高（甚至超過(guò)了 BART + DPR 和 RAG）。

結(jié)論

我們?yōu)榛?Routing Transformers 和 REALM 的長(zhǎng)格式問(wèn)答推出了一個(gè)系統(tǒng)，該系統(tǒng)在關(guān)于 ELI5 的 KILT 排行榜中名列前茅。但是，詳細(xì)的分析揭示了存在的一些問(wèn)題，即無(wú)法使用基準(zhǔn)來(lái)顯示有意義的建模進(jìn)展。我們希望社區(qū)共同合作，一起解決這些問(wèn)題，以便研究人員向正確的高峰攀登，在這個(gè)充滿(mǎn)挑戰(zhàn)但十分重要的任務(wù)中取得有意義的進(jìn)展。

致謝

Routing Transformer 是 Aurko Roy、Mohammad Saffar、Ashish Vaswani 和 David Grangier 等人進(jìn)行團(tuán)隊(duì)協(xié)作的結(jié)果。有關(guān)開(kāi)放域長(zhǎng)格式問(wèn)答的后續(xù)工作是由 Kalpesh Krishna、Aurko Roy 和 Mohit Iyyer 協(xié)作完成的。我們要感謝 Vidhisha Balachandran、Niki Parmar 和 Ashish Vaswani 提供的多條實(shí)用意見(jiàn)，感謝 REALM 團(tuán)隊(duì) (Kenton Lee、Kelvin Guu、Ming-Wei Chang 和 Zora Tung) 在代碼庫(kù)方面提供的幫助以及多條實(shí)用意見(jiàn)，這些意見(jiàn)幫助我們進(jìn)一步完善了實(shí)驗(yàn)。

我們非常感謝 Tu Vu 針對(duì) QQP 分類(lèi)器提供的幫助，這些分類(lèi)器用于在 ELI5 訓(xùn)練集和測(cè)試集中檢測(cè)解釋。感謝 Jules Gagnon-Marchand 和 Sewon Min 對(duì)檢查 ROUGE-L 邊界提供的有用實(shí)驗(yàn)建議。最后，感謝 Shufan Wang、Andrew Drozdov、Nader Akoury 以及 UMass NLP 小組的其他成員針對(duì)項(xiàng)目的不同階段提出的實(shí)用意見(jiàn)和建議。

編輯：jq