From 250ac740ff73f39f698da0b3eced713a19110ef9 Mon Sep 17 00:00:00 2001 From: Manuel Soria Date: Thu, 26 Oct 2023 17:13:18 -0300 Subject: [PATCH] incorporate lcel and other updates --- cookbook/retrieval_in_sql.ipynb | 644 +++++++++++++------------------- 1 file changed, 252 insertions(+), 392 deletions(-) diff --git a/cookbook/retrieval_in_sql.ipynb b/cookbook/retrieval_in_sql.ipynb index f167bbfe9f519..6f7ee17b35313 100644 --- a/cookbook/retrieval_in_sql.ipynb +++ b/cookbook/retrieval_in_sql.ipynb @@ -5,51 +5,57 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Incoporating semantic search in tabular databases" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In this notebook we will cover how to add embeddings into the SQL database for doing semantic search **combined** with standard tabular queries in the same solution." + "# Incoporating semantic similarity in tabular databases\n", + "\n", + "In this notebook we will cover how to run semantic search over a specific table column within a single SQL query, combining tabular query with RAG.\n", + "\n", + "\n", + "### Overall workflow\n", + "\n", + "1. Generating embeddings for a specific column\n", + "2. Storing the embeddings in a new column (if column has low cardinality, it's better to use another table containing unique values and their embeddings)\n", + "3. Querying using standard SQL queries with [PGVector](https://github.com/pgvector/pgvector) extension which allows using L2 distance (`<->`), Cosine distance (`<=>` or cosine similarity using `1 - <=>`) and Inner product (`<#>`)\n", + "4. Running standard SQL query\n", + "\n", + "### Requirements\n", + "\n", + "We will need a PostgreSQL database with [pgvector](https://github.com/pgvector/pgvector) extension enabled. For this example, we will use a `Chinook` database using a local PostgreSQL server." ] }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os\n", "import getpass\n", "\n", - "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The example will be querying over Chinook database in `PostgreSQL` with `SQLDatabase`. We will query data by combining standard SQL queries with the semantic meaninig of song titles.\n", - "\n", - "The database will need to have the vector extension enabled, so follow up the installation guide for [pgvector](https://github.com/pgvector/pgvector)" + "os.environ[\"OPENAI_API_KEY\"] = os.environ.get('OPENAI_API_KEY') or getpass.getpass(\"OpenAI API Key:\")" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/manuelsoria/miniconda3/envs/auto-gpt/lib/python3.8/site-packages/langchain/utilities/sql_database.py:112: SAWarning: Did not recognize type 'vector' of column 'title_embedding'\n", + " self._metadata.reflect(\n", + "/Users/manuelsoria/miniconda3/envs/auto-gpt/lib/python3.8/site-packages/langchain/utilities/sql_database.py:112: SAWarning: Did not recognize type 'vector' of column 'embeddings'\n", + " self._metadata.reflect(\n" + ] + } + ], "source": [ "from langchain.sql_database import SQLDatabase\n", "from langchain.chat_models import ChatOpenAI\n", "\n", - "CONNECTION_STRING = \"postgresql+psycopg2://postgres:test@localhost:5432/chinook\" # Replace with your own\n", - "db = SQLDatabase.from_uri(CONNECTION_STRING)\n", - "llm = ChatOpenAI(model_name='gpt-4', temperature=0)" + "CONNECTION_STRING = \"postgresql+psycopg2://postgres:test@localhost:5432/vectordb\" # Replace with your own\n", + "db = SQLDatabase.from_uri(CONNECTION_STRING)" ] }, { @@ -65,12 +71,12 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We will need to add a new column in the table we want for storing the embedding:" + "For this example, we will run queries based on semantic meaning of song titles. In order to do this, let's start by adding a new column in the table for storing the embeddings:" ] }, { "cell_type": "code", - "execution_count": 356, + "execution_count": 9, "metadata": {}, "outputs": [], "source": [ @@ -87,59 +93,36 @@ }, { "cell_type": "code", - "execution_count": 51, + "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "from langchain.embeddings import OpenAIEmbeddings\n", - "from tqdm import tqdm\n", "\n", "embeddings_model = OpenAIEmbeddings()" ] }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now we generate the embeddings for each song title. This can take a while, so let's limit to 200 songs" - ] - }, { "cell_type": "code", - "execution_count": 32, + "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "['Princess of the Dawn',\n", - " 'Put The Finger On You',\n", - " \"Let's Get It Up\",\n", - " 'Inject The Venom',\n", - " 'Snowballed']" + "3503" ] }, - "execution_count": 32, + "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "# Fetch the titles from the Track table\n", - "res = db.run('SELECT \"Name\" FROM \"Track\"')\n", - "titles = [title[0] for title in eval(res)]\n", - "titles[:5]" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": {}, - "outputs": [], - "source": [ - "title_embeddings = embeddings_model.embed_documents(titles)\n", - "# len(title_embeddings)" + "tracks = db.run('SELECT \"Name\" FROM \"Track\"')\n", + "song_titles = [s[0] for s in eval(tracks)]\n", + "title_embeddings = embeddings_model.embed_documents(song_titles)\n", + "len(title_embeddings)" ] }, { @@ -152,18 +135,12 @@ }, { "cell_type": "code", - "execution_count": 38, + "execution_count": 34, "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "100%|██████████| 3503/3503 [00:23<00:00, 146.68it/s]\n" - ] - } - ], + "outputs": [], "source": [ + "from tqdm import tqdm\n", + "\n", "for i in tqdm(range(len(title_embeddings))):\n", " title = titles[i].replace(\"'\",\"''\")\n", " embedding = title_embeddings[i]\n", @@ -181,23 +158,16 @@ }, { "cell_type": "code", - "execution_count": 52, + "execution_count": 21, "metadata": {}, "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "SELECT \"Track\".\"Name\" FROM \"Track\" WHERE \"Track\".\"embeddings\" IS NOT NULL ORDER BY \"embeddings\" <-> '[-0.005296176500367851, -0.024523600003614955, -0.0007927474517293692, -0.01893319261373208, 0.0035659644242875383, 0.002227528338073427, -0.00032501444871436925, 0.0019508862909545174, -0.017282933652393022, 0.008219307600334083, 0.006239637969211901, 0.009786413422444112, -0.022975684083882866, -0.00395933972790567, 0.012645581285559251, 0.007451745964153147, 0.03661269833460187, -0.01778184876247676, 0.0186133745668314, -0.013918454355509036, -0.0026528854929477575, 0.0038154219793294078, 0.01217225147021595, -0.00718309943805595, -0.006217250836314019, -0.014506918928506913, 0.021440560811520994, -0.026992590259293216, 0.02301406202599352, -0.022898928199661557, 0.018626167214201617, -0.010643523590216577, -0.03175147370746667, -0.017206177768171713, -0.0012264998396480686, -0.00873101535778804, 0.0011041696835090498, 0.0010410057542879437, 0.012747922464520998, -0.01273512981715078, 0.005097890000468161, -0.006901660264588535, 0.002987094802478629, -0.025534222871152652, -0.018843642219495332, 0.013112514777217444, 0.0024386079510046415, 0.004416678734036199, -0.015120967864922619, 0.012025135094135827, 0.02334667272026442, 0.020621827654536573, -0.01829355651993072, -0.035870721061838765, 0.013176478014068536, 0.0290650084464947, -0.0221953288690091, 0.005161853237319253, -0.013496295129646608, -0.01724455571028237, 0.019636790081739314, 0.0026976599915741746, -0.010675506139964732, 0.03326101354773332, 0.0016550551563649536, 0.009025248109948284, -0.022553524858020438, 0.009153174583650468, 0.010886584821573337, 0.010029474653800871, 0.0290650084464947, 0.02704376084877409, 0.0031246162273697825, -0.0004405485109379595, 0.007125532524889967, 0.012524050204219567, -0.022016231805826042, -0.008174533334538318, -0.011993153475710281, 0.00898047384415252, 0.016285103432225546, -0.02213136563215801, -0.02489458863999651, -0.023973514304050345, -0.004730099991590466, 0.007477331258893584, -0.024267747056210588, 0.01347070983490617, -0.016285103432225546, -0.026109895728102923, 0.02816951940528897, 0.009133985612595142, 0.00473329815343302, 0.021913890626864295, -0.00831525338693333, 0.0051298716188937065, 0.005529643246196948, 0.0159908706800653, 0.0060317565181232415, -0.02563656591275962, -0.005369734688407912, 0.00026145074926294394, -0.015722225085290715, -0.0007583670955065809, -0.013368368655944422, -0.018574996624720744, 0.023576940372928352, 0.0019109091515072585, 0.03428442906264124, -0.025252784629007846, -0.013624222534671401, 0.03796872640642591, 0.004554200158927353, -0.027222859774602367, 0.008839753791757507, -0.029218520214937323, -0.011046493845023677, 0.003818620141171962, 0.019214630855876887, 0.01755158110981283, 0.00603815284180835, 0.0076308439586588155, 0.038736285248639014, -0.010413255006230035, -0.003250944085811994, 0.020199668428674145, -0.03689414030203712, -0.0014207888700752186, -0.02154290199048274, 0.010841810555777572, 0.010304517503583178, -0.007733185603281868, 0.011340725665861312, -0.015530334443414827, -0.026787907901369718, 0.01829355651993072, -0.004100059314639378, -0.008053003184521243, -0.01233216049366629, -0.02113353541199053, 0.01029172485621296, 0.04326490104414854, 0.008884528057553272, 0.009767224451388784, -0.021530109343112523, -4.297546710635903e-05, 0.009933528867201624, 0.011334329342176201, 0.001173729936415265, -0.0006612225220853428, 0.0212486692383225, -0.03006283866666218, 0.0036235315702841738, 0.006850489209446356, -0.000268246872282204, 0.004170419340836885, -0.0025089677443714954, 0.00482284668502455, -0.03205849724435191, 0.0075796729035166365, 0.008436783536950408, -0.022758209078589154, 0.004973161222947226, 0.004621362023282304, 0.0059294153391614935, 0.026813493196110157, 0.0158501515589929, 0.017039873352358872, -0.01347070983490617, 0.009536955867402242, -0.024203783819359496, 0.016643299421236882, -0.029192934920196884, -0.0055232469225118385, -0.012581618048708158, 0.016643299421236882, -0.00514586242810648, 0.020046156660231525, -0.027862494005758506, -0.015555919738155264, -0.02352576978344748, 0.014097552350014704, 0.01648978579014904, 0.012696751875040125, -0.009524163220032024, 0.011532616307737198, 0.015555919738155264, -0.0015399208643716827, -0.004762081610016012, -0.009939926122209343, 0.02334667272026442, 0.0020212462007366976, 0.01991823018652934, -0.012396123730517381, -0.6906009578429887, -0.028527715394300306, 0.007905886808441122, -0.012191440441271276, 0.012760715111891217, -0.000906682433031682, 0.002566534890368131, -0.013982417592360127, -0.02671115201714841, 0.003892178096381371, -0.0006228444635593613, 0.025176028744786538, -0.022732621921203497, -0.0017957748595139875, 0.010387669711489599, -0.016732847952828412, 0.0017062259786764797, -0.047384148398520634, 0.0035339825730313396, -0.01297819104850754, -0.008974076589144802, 0.01656654353701557, -0.020954438348807475, -0.02328270948341333, 0.018012118277785913, -0.003927358109480124, 0.005049917107168536, -0.015798980969512023, -0.023717661356645974, -0.0005924618096397662, -0.0029071405235841113, 0.03689414030203712, -0.006722562270082866, -0.014877906633565858, 0.04968683610103127, 0.011404689834035014, -0.02048110853346417, 0.01853661868261009, 0.01591411479584399, 0.05047998396327526, -0.011155232278993144, -0.0005444891491707945, 0.0036747021597650473, 0.011916397591488971, -0.0011161627904186296, 0.014263857697150153, 0.04377661066424772, -0.005248204072729532, 0.0018773283357451087, -0.007522105990350653, 0.025521430223782435, -0.0037738456425455455, 0.0018165630279059187, 0.009760828127703675, -0.006300403509881743, -0.0017158204642041435, 0.03218642744334454, -0.012805489377686982, 0.018306349167300937, 0.02431891764569146, -0.00799543534003265, 0.012037927741506046, -0.020455523238723735, -0.004141635884253893, -0.016604921479126226, 0.016771225894939065, -0.006018963870753023, -0.009875961954035641, 0.016297896079595763, -0.002860766944036417, -0.011136043307937816, 0.01568384714318006, -0.008021021566095698, -0.0064571139058282245, 0.029934912192959993, 0.02132542698518903, 0.0010961742789026632, -0.035973065966090956, -0.01299098369587776, 0.031470035465321865, 0.012927020459026667, -0.004931584653332711, -0.016502578437519257, -0.001415192086850748, 0.03280047451711502, 0.0022579111084083483, -0.014199893528976451, -0.0028527715394300306, 0.0033836685007699683, 0.012959002077452212, 0.014519711575877132, 0.00391776362395246, 0.0007407771471648674, -0.02809276352106766, 0.03661269833460187, 0.010151004803817946, -0.015862944206363118, -0.0002236723169786098, 0.010976134284487475, 0.002971103993265856, -0.0013624223000332706, -0.02228487926324585, 0.02089047511195638, 0.03139327958110055, 0.026173858964954014, -0.01134712198954642, 0.011321536694805984, 0.0015846953629980996, 0.014199893528976451, -0.04303463339148461, -0.014813943396714766, -0.0093066872834157, -0.00955614483845757, -0.007406971698357383, 0.01199954979939539, -0.027888079300498945, 0.006108512868005857, 0.014340613581371463, -0.007752374574337195, -0.01949607096066691, 0.0349496467258926, 0.0004805256794890499, 0.01696311560549234, -0.010278932208842741, 0.01061793829547614, 0.006373961232260499, 0.013982417592360127, -0.023551355078187914, -0.024075855483012092, -0.007490124371925108, 0.00482284668502455, -0.012057116712561374, 0.021376597574669903, 0.002568133971289408, 0.008954887618089474, -0.004016907106732958, 0.0008403202837603583, -0.012434501672628037, 0.00015131361425249615, -0.0015782990393129904, -0.01779464140984698, 0.017078251294469528, -0.0012033130498742214, 0.015069797275441745, -0.004259968338089718, -0.02106957217513944, -0.008788583202276634, -0.018408690346262687, -0.018818056924754894, 0.007387782727302055, -0.0064187359637175685, -0.0055968051105519, -0.01724455571028237, 0.008935698647034146, 0.0007575675550459423, -0.01568384714318006, 0.0015175337314738003, -0.0065626537122938314, -0.018152837398858316, -0.007279044758993894, 0.013649807829411838, 0.01911228967691514, 0.006805714943650592, -0.00371627849654891, 0.002059624142847353, -0.00988235827772075, -0.009613711751623552, 0.03349128120039726, 0.02034038754974655, -0.015952492737954647, -0.00538892365946324, -0.036970896186258435, 0.005491265304086292, 0.02589241886016399, 0.016911945016011468, 0.009652089693734208, -0.01224900735443726, -0.007899490484756012, 0.012357745788406727, -0.001321645503710047, -0.007707599842880126, -0.018869227514235767, -0.022745416431218933, -0.002297888131440281, 0.00279840255527595, 0.004957169948073148, 0.02048110853346417, 0.008545521970919873, 0.0013536272385509193, 0.01577339567477159, -0.017602751699293703, 0.024382880882542552, -0.007534898637720873, -0.011692525331187538, -0.003017477339982898, -0.004541407511557133, -0.0009706459027134267, 0.007368593756246727, 0.03226318332756585, 0.00010613941087938083, 0.029755813267131715, -0.01444295476033321, 0.02564935856012984, -0.01797374033567526, -0.012357745788406727, -0.02914176433071601, 0.01126396978164, -0.012338556817351399, -0.0034508301322942673, -0.024037477540901436, -0.001647059751758567, -0.040194654499424796, -0.018012118277785913, -0.016361859316446858, 0.005475274494873519, 0.018255178577820063, 0.01485232133882542, 0.014020796465793392, -0.02645529720709882, -0.011679732683817319, 0.015645469201069403, 0.00041176493793964176, -0.011116853405559879, 0.004362309517051466, -0.0020884078322609967, 0.01069469511102006, 0.012344953141036508, 0.013496295129646608, 0.008865339086497944, -0.008174533334538318, -0.03674062480830406, 0.0036938913636510276, -0.0033101103127299073, 0.026762322606629283, 0.014110344997384922, -0.016208347548004234, 0.02618665161232423, -0.033772719442542065, 0.017666714936144797, -0.0075796729035166365, 0.012952605753767103, -0.014263857697150153, 0.025457466986931344, 0.02648088250183926, -0.011782073862779067, 0.014020796465793392, 0.022054609747936698, 0.006824903914705919, -0.021376597574669903, 0.0005189037962226946, -0.019073911734804484, -0.00657544635966405, 0.0045062277312890335, 0.003073445405058256, 0.01444295476033321, -0.023576940372928352, -0.007650032929714143, -0.005289780176682742, 0.042932292212522864, 0.0326469627486724, 0.007656429253399252, 0.009268309341305046, 0.01363701518204162, 0.004570190968140125, -0.009920736219831404, -0.012466483291053583, -0.015325651154168724, -0.0034604246178219313, -0.045337317368704805, 0.005600003272394454, -0.009114796641539814, -0.020225253723414584, 0.023065232615474395, -0.021760376995776456, 0.00482284668502455, 0.0008067394679982087, -0.0047652797718585665, -0.008046606860836135, -0.0043814989537680985, 0.005040322621640872, -0.011308744047435764, -0.01911228967691514, 0.009377046843951901, 0.016349066669076637, -0.0039657360515907795, -0.013304405419093331, -0.009978304064319998, 0.020122912544452837, -0.00939623581500723, 2.9208325230053062e-05, 0.00543049976341645, 0.022054609747936698, 0.009811998717184548, 0.0005936611203307242, -0.007387782727302055, -0.009364254196581684, 0.01354746665045009, 0.00779714884013296, 0.016617714126496444, -0.023858380477718377, -0.01608042107430205, -0.0018981163877217137, -0.023640903609779444, -0.01672005530545819, 0.006658599033231774, 0.012300177943918134, 0.006639409596515142, 0.002205141205175545, -0.0007435755387771027, 0.010637127266531467, -0.012229818383381932, -0.020519486475574826, -0.00047293001600915113, -0.0011921194834252802, -0.010400462358859816, -0.015735017732660932, -0.021824342095272765, 0.0069464345303842994, 0.016860774426530594, 0.012920624135341558, -0.01700149541024822, -0.020621827654536573, -0.02760664105835414, 0.015658261848439624, 0.10474660644929693, 0.0028831540769342994, -0.008423990889580188, 0.011737299596983303, 0.012722337169780561, 0.009575333809512898, 0.007023190880266915, -0.049405397858886466, -0.009914339896146296, -0.0018645355719595642, 0.017525995815072394, -0.005385725497620685, 0.0027088535580231158, -0.012703148198725233, 0.01877967898264424, 0.011788470186464176, -0.01118081757373358, 0.02393513636193969, -0.018267971225190284, -0.008475161479061062, -0.01273512981715078, 0.006997605119865173, -0.009920736219831404, 0.032723718632893715, 0.02996049748770043, 0.020455523238723735, 0.012645581285559251, 0.0241526113672334, 0.014954662517787168, -0.011711714302242866, 0.0031198189846059505, 0.01020217632462143, -0.009242724046564609, -0.0024274143845557004, -0.02375603929875663, 0.016950322958122124, -0.005494463465928847, -0.0035659644242875383, 0.026378541322877508, -0.006722562270082866, 0.008782186878591525, 0.03945267722666169, 0.012709544522410344, -0.023730454004016195, 0.02089047511195638, 0.003908169138424796, -0.014430162112962993, 0.024357295587802117, -0.007406971698357383, -0.018229593283079628, 0.03267254804341284, 0.005926217177318939, -0.024549185298355394, -0.0034284427665657326, 0.007144721495945295, 0.02122308394358206, -0.007694807195509907, -0.008929302323349037, -0.006908056588273644, -0.0023042844551253903, -0.017167799826061057, -0.04590019757828486, 0.02270703662646306, -0.006594635330719377, -0.004666136289078068, -0.017039873352358872, -0.03986204566779912, -0.00882056482070218, -0.03571720929339615, 0.0024817831358791287, 0.0017877794549076011, -0.012114684557049966, -0.015210517327836758, -0.00632598880462218, 0.018869227514235767, 0.014788358101974328, -0.015287273212058068, 0.010976134284487475, -0.002341063549145421, 0.008641466826196511, 0.002446603355611028, -0.007899490484756012, -0.004240778901373085, 0.011954775533599625, -0.025201614039526973, 0.01248567226210891, -0.01755158110981283, -0.029423202572860817, 0.0013880075947737074, 0.012799093054001872, 0.01535123644890916, 0.005350545717352584, 0.024037477540901436, -0.009818395040869657, 0.01591411479584399, -0.030088423961402613, 0.020775339422979197, -0.008065795831891463, -0.0012272993801087072, -0.00996551141694978, -0.010432444908607973, -0.006255628778424674, -0.016950322958122124, -0.021760376995776456, 0.014404576818222556, 0.026327370733396634, -0.00029383225433413557, 0.02834861646847203, -0.0010625934631405137, -0.01380332052917707, 0.03625450420823576, -0.040194654499424796, 0.021811549447902548, 0.009594522780568224, -0.01803770357252635, 0.0027632225421771964, -0.0047556852863309025, 0.0182807638725605, 0.0017861803739863237, -0.01803770357252635, -0.0025297557963481, -0.008059399508206352, 7.250860928652549e-05, -0.00035299848125204834, 0.009377046843951901, 0.005395319983148349, 0.005609597757922118, -0.013700978418892711, -0.006888867617218316, 0.006620220625459815, -0.005983784090484922, 0.02980698385661259, -0.007515709666665545, -0.03203291194961148, -0.004221589930317758, -0.01991823018652934, -0.003978528698960998, -0.009978304064319998, 0.011168024926363361, -0.00534734755551003, -0.008878131733868163, -0.018421482993632905, -0.001696631493148816, -0.02141497551678056, 0.026557638386060567, -0.03233993921178716, -0.003700287687336137, 0.008775790554906415, -0.001511137291373365, 0.008283271768507785, -0.013080533158791898, -0.006338781451992398, -0.03175147370746667, 0.016873567073900815, -0.0006192464732788242, -0.024139818719863183, 0.006792922296280373, -0.034565867304786044, 0.0019157063942710905, 0.01755158110981283, 0.018728508393163368, 0.004749288962645794, 0.009639297046363989, 0.0056191922434497815, 0.010649919913901686, -0.005558426702779939, -0.0019668772165826165, -0.011993153475710281, -0.02360252566766879, 0.0342588437679008, -0.004077672181741495, 0.01935534997694929, 0.015274480564687849, -0.025585393460633526, 0.001475957394689938, 0.009313083607100809, -0.0044838401327298456, -0.0019268999607200315, -0.017282933652393022, -0.02301406202599352, -0.03139327958110055, -0.004925188329647601, -0.026506467796579693, -0.0054177071160462316, 0.0020708179421269463, -0.0009146778376380686, 0.014161515586865795, 0.0026704756159124606, 0.012293781620233025, -0.007176703114370841, 0.011890812296748534, -0.007496520695610217, 0.02891149481540686, 0.0029119377663479433, -0.007451745964153147, -0.022898928199661557, 0.00017300123655881428, -0.011532616307737198, 0.0027680197849410284, 0.01428944299189059, 0.016975908252862562, 0.01574781038003115, -0.005772704477553708, 0.007400575374672274, -0.004160824855309221, 0.007074361469747789, -0.03384947532676338, -0.001045003456591137, -0.009133985612595142, -0.010502804469144174, 0.00206122322376863, 0.007029587203952024, -0.013368368655944422, -0.035896306356579206, 0.02824627528951028, -0.014468540055073647, -0.0337983047372825, 0.028655641868002492, -0.007368593756246727, -0.016975908252862562, -0.015223309975206975, -0.01249206858579402, 0.010662713492594515, 0.012204233088641495, 0.020135705191823054, 0.021504524048372085, 0.002301086293282836, -0.03454028201004561, 0.006018963870753023, 0.004455056676146854, 0.01876688633527402, 0.018587789272090965, -0.0015095382104520876, 0.012127477204420185, -0.02222091416374954, 0.009281101988675263, 0.040373751562607856, -0.019956608128639995, -0.02245118367905869, 0.0037770438043881, 0.004947575462545484, 0.018843642219495332, 0.006581842683349159, -0.003671503997922493, -0.015978078032695082, 0.03707323736522018, 0.008232100247704302, -0.002022845281657975, -0.016732847952828412, -0.03781521091269285, -0.01427665034452037, 0.014315028286631027, -0.021837134742642986, 0.025278369923748285, 0.002074016103969501, -0.011129646052930098, -0.002568133971289408, -0.0046085689102507805, -0.005942207986531712, -0.001412793465468832, 0.004710910554873833, 0.033823890032022935, -0.002561737647604299, -0.01642582255329795, -0.013726563713633148, -0.027555470468873265, -0.017078251294469528, -0.004442264028776636, 0.026608810838186663, 0.019189045561136448, -0.040194654499424796, -0.0017941757785927103, -0.004138437722411338, -0.006728958593767976, 0.004842035656079877, -0.0124089163778876, -0.008769394231221306, -0.0019620799738187845, -0.02443405147202343, -0.0063227906427796255, 0.018498240740499435, 0.003994519508173771, -0.008360027652729096, -0.011807659157519504, 0.010515597116514392, -0.008974076589144802, -0.007534898637720873, 0.002563336728525576, -0.018907605456346423, 0.002547345919312803, -0.022118572984787793, 0.008353631329043987, 0.011737299596983303, 0.019444900371186038, -0.0003783840072926518, 0.001478356016071854, 0.00200045791592944, 0.02648088250183926, -0.023474599193966605, 0.01199954979939539, 0.004829243008709658, 0.010240554266732085, -0.00030502587899073984, 0.027222859774602367, -0.020302009607635892, -0.03592189537661008, -0.004106455638324487, -0.020788132070349415, -0.005094691838625606, 0.003047860110317819, 0.012319366914973462, 0.019547241550147785, -0.0020804124276546103, -0.0028191906072525547, -0.01208909833098692, 0.021210291296211842, -0.005743921020970716, -0.017180592473431275, -0.03927358016347863, 0.03387506062150381, 0.008264082797452457, -0.010099834214337073, -0.00825768554244474, -0.021363804927299685, 0.030548961129375697, 0.005750317344655826, -0.002598516508793677, 0.010918567371321493, -0.03356803708461857, -0.02301406202599352, 0.003508397278290903, 0.021440560811520994, -0.011948379209914517, -0.02222091416374954, 0.011481445718256324, 0.008878131733868163, -0.005069106078223864, 0.004576587291825234, 0.007950661074236887, 0.004196004635577321, 0.012140269851790403, 0.00783552724790492, 0.004528614864186915, -0.011654147389076882, -0.0032637369660128653, -0.01551754179604461, 0.0005664766282536838, -0.02350018448870704, 0.0022435191472862004, -0.011820451804889722, 0.0024242162227131455, 0.01028532853252785, 0.003988123184488662, -0.023295502130783546, -0.01779464140984698, -0.013253233898289847, -0.023001269378623304, -0.004707712393031279, 0.003447631970451713, 0.01576060302740137, 0.024523600003614955, 0.004963566271758257, 0.0046085689102507805, 0.027657811647835012, -0.016106006369042487, 0.013496295129646608, 0.0011825249978976164, -0.006233241645526791, 0.01282467928006492, -0.015798980969512023, 0.0023890362096143924, 0.025291162571118502, -0.027222859774602367, -0.0047556852863309025, 0.03940150663718081, 0.018063288867266786, 0.004327129736783365, 0.01779464140984698, -0.0021683621111555145, 0.011487842041941433, 0.011091268110819442, -0.015479163853933954, 0.0016886359721271032, 0.0028175915263312775, 0.011602975868273399, -0.025598186108003743, 0.008308857063248222, 0.014967456096479996, -0.03610099243979314, -0.002793605312512118, -0.009172363554705796, 0.013624222534671401, -0.007937868426866668, 0.004474245647202182, -0.03436118494686255, -0.0028943476433832406, -0.003294119503517134, -0.016349066669076637, 0.014813943396714766, -0.01077784731892648, -0.006297205348039189, -0.002863965105878972, 0.02393513636193969, -0.016208347548004234, 0.00898687016783763, 0.005251402234572087, -0.015312858506798505, -0.0205706570650557, 0.01020217632462143, -0.0323655245065276, 0.013445124540165733, -0.0018101665878054833, 0.007496520695610217, -0.021261461885692716, 0.008123362745057445, -0.00914677825996536, -0.012127477204420185, -0.016221140195374455, 0.0012888642284085358, 0.0302163504351048, -0.02390955106719925, 0.002358653439279471, 0.0059581987957444855, -0.013892869060768599, -0.0063963483651583814, -0.027222859774602367, 0.004960368109915702, -0.011334329342176201, -0.025009722466328477, -0.007035983527637134, 0.00015561117333226046, -0.021504524048372085, -0.014570882165358005, -0.014890699280936077, 0.020967230996177692, 0.01642582255329795, 0.2327247499298811, -0.03257020686445109, -0.01437899152348212, 0.004378300791925544, 0.02776015282679676, 0.01877967898264424, 0.02939761727812038, 0.0010553974825794395, 0.003911367300267351, 0.009901547248776078, -0.024715491576813454, 0.0023090819307198746, -0.004905999358592274, -0.007701203519195017, -0.025930796802274645, -0.02237442779483738, -0.03899214192133382, -0.011404689834035014, -0.03840367641701334, -0.01501862668596087, 0.005395319983148349, 0.003994519508173771, 0.002062822304689907, -0.008929302323349037, 0.011078475463449223, -0.0002692462978580023, -0.0010601947253432713, 0.0020804124276546103, 0.006252430616582119, 0.005587210625024236, -0.0158501515589929, -0.012242611030752151, -0.0016486588326798442, 0.006837696562076137, -0.012703148198725233, -0.02155569463785296, 0.02801600763684635, -0.0002796403529501364, 0.023692074199260317, 0.0006100517579814797, 0.015466371206563736, 0.009658486017419317, -0.003254142247654549, 0.004346318707838693, 0.0018965173068004365, 0.032544621569710656, 0.0070999467644882255, -0.012562428146330221, 0.007355800643215204, -0.0024945760160799994, -0.03914565368977645, -0.01692473766338169, 0.04308580398096548, 0.024945759229477386, -0.009722450185593019, -0.0007915481410384112, 0.030395447498287858, 0.016911945016011468, -0.007221477380166605, 0.00018619371236701517, -0.015786188322141806, 0.012338556817351399, -0.0075988618745719645, 0.036459186566159256, -0.010797036289981807, 0.02498413717158804, -0.0008739010995225079, 0.013764941655743804, 0.029116179035975576, 0.007790752516447851, -0.002067619780284392, -0.005625588567134891, -0.01159018322090318, 0.00640594331634735, -0.02475386951892411, -0.014877906633565858, 0.033951816505725124, 0.003524388087503676, 0.005539237731724611, 0.0047556852863309025, -0.016208347548004234, -0.02188830533212386, 0.0018549410864319004, -0.022796587020699807, -0.005836668180066105, -0.03530784457754915, 0.014097552350014704, 0.022579110152760876, -0.009165967231020688, -0.006620220625459815, 0.013253233898289847, -0.024523600003614955, 0.014315028286631027, -0.005980585928642368, 0.009377046843951901, 0.03658711303986144, 0.01299098369587776, 0.036510357155640126, -0.012031531417820936, 0.02287334290492112, -0.009038040757318504, -0.03062571701359701, -0.012293781620233025, 0.01427665034452037, -0.007496520695610217, 0.0003194176654898987, -0.011973964504654953, 0.012831075603750028, -0.004551001997084797, -0.01879247163001446, 0.0011697322341120717, -0.024382880882542552, 0.004394291601138317, 0.011129646052930098, 0.000911479675795514, -0.0012009144284923054, 0.0059294153391614935, -0.011577390573532962, 0.007816338276849593, -0.016707262658087974, 0.012504861233164239, -0.017602751699293703, 0.0003306112610426714, 0.017948153178289603, -0.010010285682745545, -0.0014503719835341749, -0.01876688633527402, 0.011481445718256324, 0.001914107313349813, -0.040885461182707035, 0.0171294218839504, -0.0022898927268338947, 0.011609372191958508, -0.011174421250048472, 0.011168024926363361, 0.007176703114370841, -0.024242161761470152, -0.014263857697150153, -0.013726563713633148, 0.0178074340572172, -0.028655641868002492, -0.024485222061504303, 0.004100059314639378, 0.011666940036447101, 0.014954662517787168, -0.008942094970719254, 0.03308191648455027, -0.009760828127703675, -0.014187100881606232, -0.001473558773308022, -0.021990646511085607, -0.012140269851790403, -0.010809828937352026, -0.016272310784855328, 0.019009948497953393, -0.014711601286430408, -0.045107049716040876, -0.020302009607635892, 0.009268309341305046, 0.01683518913179016, -0.025495844929041996, -0.011705317978557757, -0.0017797839338858882, -0.013496295129646608, -0.019956608128639995, -0.005337753069982366, -0.16405355062563687, 0.021299839827803372, 0.017999325630415695, -0.014008003818423174, 0.012338556817351399, -0.0017510004773028968, 0.0287323977522238, 0.008801375849646851, 0.00142958393155757, 0.005091493676783051, 0.0007335812248114564, 0.016272310784855328, -0.044851196768636505, -0.02284775761018068, 0.024267747056210588, 0.0256237732653894, -0.013496295129646608, 0.03709882265996062, 0.01721897041554193, 0.004087266667269159, -0.0014615656663984422, -0.011219195515844237, 0.025329540513229158, 0.007931472103181559, 0.010573164029680376, 0.011046493845023677, 0.0033069121508873525, 0.01273512981715078, 0.00024386078636931468, -0.006185269217888472, -0.017014288057618437, -0.004522218540501806, 0.04677010132475015, 0.0314956207600623, -0.026608810838186663, 0.0009962312556615267, -0.013521880424387045, -0.019137874971655575, -0.02287334290492112, 0.017935360530919382, 0.024485222061504303, 0.022054609747936698, 0.0005532841524454829, -0.004905999358592274, -0.019368142624319507, 0.0021635648683916825, 0.016784018542309286, 0.004362309517051466, 0.0038506017595975085, 0.004909197520434829, 0.015312858506798505, -0.009811998717184548, 0.004218391768475204, 0.008699033739362495, 0.018600581919461182, -0.0033037139890447977, 0.0007879502089655372, 0.004778072419228785, -3.832811998900214e-05, -0.014110344997384922, -0.014647638049579316, -0.013931247002879253, -0.005692750431489843, -0.009178759878390905, -0.01324044125091963, -0.004509425893131588, -0.009754431804018566, -0.005641579376347664, -0.031649132528504924, 0.004845233817922431, 0.011027304873968349, 0.014801150749344547, -0.006677788004287102, -0.005843064503751215, 0.028041592931586784, 0.00019099096968276286, -0.01982867979229259, 0.017500410520331956, -0.004423075057721308, -0.013867283766028162, -0.00986956563035053, 0.01281828295637981, -0.007023190880266915, 0.01494186987041695, 0.006754543888508413, 0.009741639156648347, 0.027888079300498945, -0.010713884082075388, -0.014315028286631027, -0.012933416782711775, 0.038557188185455955, -0.02929527609915863, -0.0030846392043378497, 0.005344149393667476, 0.0166560920686071, 0.016950322958122124, 0.0012025135094135828, 0.0060317565181232415, 0.011289555076380438, -0.011027304873968349, 0.010726676729445606, 0.006549861064923613, -0.01111045708187477, 0.00555203037909483, -0.008858942762812835, 0.016272310784855328, 0.025188821392156755, 0.02378162459349707, 0.021210291296211842, -0.010029474653800871, -0.013739356361003367, 0.004036096077788286, 0.031163010065791405, 0.005021133650585545, 0.013892869060768599, 0.020545071770315265, -0.009760828127703675, -0.0067161659463977575, 0.004148032207939002, -0.0007435755387771027, 0.018920399966361863, 0.017193385120801496, -0.016042043132191396, 0.0047141087167163875, -0.007016794556581806, -0.012466483291053583, -0.09251678340930194, -0.03390064591624425, 0.014558089517987788, 0.028655641868002492, 0.0037802419662306547, 0.039017727216074265, -0.003332497678458442, 0.0005768707124496493, -0.03766169914425023, 0.023129195852325486, -0.009524163220032024, -0.011551805278792526, -0.00222592925715215, 0.00979920606981433, 0.02098002364354791, -0.0243445029404319, -0.017692300230885232, -0.0027184480435507798, -0.026250614849175326, 0.012280988972862806, -0.012645581285559251, 0.002030840686264361, -0.0020068544724452016, -0.002026043443500529, -0.007035983527637134, -0.00037018865936344245, -0.0006320391788567057, 0.02456197794572561, 0.009709657538222801, 0.008059399508206352, -0.0006596234411640655, 0.0006788124704270562, 0.009038040757318504, -0.014609260107468661, -0.0028527715394300306, -0.01788418994143851, -0.04121807001433271, -0.01020217632462143, 0.00907002237574405, -0.02694141966981234, 0.000906682433031682, 0.004167221178994329, 0.010074248919596636, -0.034821720252190415, 0.0004945176957578894, -0.01696311560549234, -0.018485448093129218, 0.00391776362395246, 0.0058878387695469785, -0.022489561621169347, -0.015338443801538941, -0.004384697115610653, -0.025265577276378067, -0.015722225085290715, 0.012952605753767103, -0.010822621584722244, -0.007515709666665545, 0.029986082782440866, -0.007074361469747789, 0.015389614391019815, 0.004742892638960684, 0.0010090240194470714, -0.017999325630415695, 0.018664545156312273, 0.002841577740150437, 0.013176478014068536, -0.01485232133882542, -0.023871173125088595, 0.026813493196110157, -0.010355688093064051, -0.0005572818547486761, 0.03060013171885657, -0.023039647320733957, 0.005193834855744799, -0.003242948681205608, 0.013841698471287725, -0.00792507577949645, -0.036024236555571826, 0.024779454813664545, -0.014071967055274267, -0.02000777871812087, -0.01591411479584399, 0.014993041391220433, -0.01574781038003115, 0.023564147725558135, 0.0070999467644882255, 0.0166560920686071, -0.003853799921440063, 0.01559429768026592, -0.0006616222923156621, 0.0009018851320601871, 0.012556031822645112, 0.0025377512009544865, -0.016847981779160377, 0.007087154117118007, 0.009006059138892956, 0.004429471381406418, -0.007240666351221933, 0.010605145648105921, 0.017577166404553268, -0.0031358097938187232, -0.019662375376479753, -0.020711376186128103, 0.03187940018116885, 0.012613599667133706, -0.010048663624856199, -0.013125307424587663, -0.0023490589537518074, -0.0004485439446481776, -0.023449013899226167, -0.008142551716112773, 0.000906682433031682, 0.001858139248274455, 0.0054081126305185676, 0.015543127090785046, 0.004989152032159999, -0.03144445017058143, -0.0050595115926962, 0.024357295587802117, 0.010745865700500934, 0.0017589958819092834, 0.007502917019295326, 0.009255516693934826, -0.006134098162746294, 0.017410860126095207, 0.007266252111623675, -0.03182822959168798, 0.013329990713833768, -0.014570882165358005, 0.009351461549211465, -0.006348375937520062, -0.014404576818222556, 0.01307413683510679, -0.01796094768830504, -0.011718110625927975, 0.02156848728522318, -0.0032669351278554197, -0.01322764860354941, 0.009038040757318504, 0.025905211507534207, 0.02295009878914243, 0.020800924717719632, -0.005900631416917198, -0.011858830678322987, 0.011973964504654953, -0.0016982305740700933, 0.02277100172595937, -0.006361168584890281, -0.047332977809039764, 0.013445124540165733, 0.017103836589209966, -0.020621827654536573, 0.01363701518204162, 0.02195226856897495, -0.008673448444622058, -0.01044523755597819, -0.03443794083108386, -0.024766662166294327, 0.0003306112610426714, 0.024421258824653208, -0.0010034272362226008, -0.012242611030752151, 0.006773733325225045, -0.013483502482276389, 0.010797036289981807, 0.004036096077788286, 0.03036986220354742, 0.01762833699403414, 0.026225029554434887, -0.011500634689311652, -0.016464200495408605, -0.001478356016071854, -0.022681451331722623, -0.002018048038894143, 0.009978304064319998, 0.03876187054337945, -0.0037802419662306547, 0.002139578654572523, -0.016131591663782925, -0.007573276579831528, -0.023244331541302673, 0.01217225147021595, 0.015287273212058068, -0.015210517327836758, -0.013944039650249472, 0.017525995815072394, 0.0045989744247231165, 0.033772719442542065, 0.0007607657168884969, 0.011052890168708786, 0.009121192965224923, 0.017321311594503678, -0.009114796641539814, -0.003661909512394829, -0.003671503997922493, 0.01494186987041695, -0.0031310125510548917, 0.01192279391517408, -0.004467849323517073, -0.010905774723951274, -0.006377159394103054, 0.020519486475574826, 0.002369847238559065, -0.00656905003597894, -0.011411086157720123, -0.018754093687903803, -0.009978304064319998, 0.01020857264830654, -0.02996049748770043, 0.017180592473431275, 0.00882056482070218, 0.01322764860354941, 0.011622165770651336, 0.018830849572125115, 0.017858604646698074, 0.010579560353365484, -0.018395897698892466, 0.009287498312360372, 0.01412313764475514, -0.011129646052930098, -0.018626167214201617, 0.03077922878203963, 0.017014288057618437, 0.024869003345256074, 0.014545296870617569, -0.01920183820850667, 0.0028783568341704674, 0.0013184473418674924, 0.014340613581371463, -0.020532279122945047, 0.0027376372474367597, 0.017346896889244116, 0.0028911494815406858, -0.008059399508206352, -0.02049390118083439, -0.013892869060768599, 0.003169390725996199, -0.011852434354637878, 0.012645581285559251, 0.03428442906264124, -0.003920961785795015, 0.03525667398806828, 0.026992590259293216, -0.015120967864922619, 0.00694003820669919, 0.010342895445693834, 0.004953971786230593, -0.009332272578156137, 0.030983913002608346, -0.0005852658872863552, -0.015466371206563736, 0.01574781038003115, -0.009351461549211465, 0.0202380463707848, -0.03454028201004561, -0.007643636606029034, -0.0031661925641536448, -0.0012496866294219157, 0.002513764987135327, -0.009933528867201624, 0.00019828680548992212, 0.005184240370217135, -0.006901660264588535, 0.014622052754838879, 0.01550474914867439, -0.008142551716112773, -0.004138437722411338, -0.00518743853205969, -0.009293894636045482, -0.00906362605205894, -0.019470485665926476, -0.019623997434369097, 0.0016038844504687535, -0.003052657353081651, -0.015325651154168724, 0.02278379437332959, 0.0012632788172527729, 0.008590296236715637, -0.028118348815808096, -0.005683155480300874, 0.013432331892795515, -0.010087041566966855, -0.0063323851283072895, -0.0070807577934328975, -0.028297445878991155, 0.01028532853252785, 0.00831525338693333, -0.013777735234436633, 0.011570994249847854, -0.009121192965224923]' LIMIT 5\n" - ] - }, { "data": { "text/plain": [ "'[(\"Tomorrow\\'s Dream\",), (\\'Remember Tomorrow\\',), (\\'Remember Tomorrow\\',), (\\'The Best Is Yet To Come\\',), (\"Thinking \\'Bout Tomorrow\",)]'" ] }, - "execution_count": 52, + "execution_count": 21, "metadata": {}, "output_type": "execute_result" } @@ -205,7 +175,6 @@ "source": [ "embeded_title = embeddings_model.embed_query(\"hope about the future\")\n", "query = 'SELECT \"Track\".\"Name\" FROM \"Track\" WHERE \"Track\".\"embeddings\" IS NOT NULL ORDER BY \"embeddings\" <-> ' + f\"'{embeded_title}' LIMIT 5\"\n", - "print(query)\n", "db.run(query)" ] }, @@ -222,56 +191,71 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Now let's try to generate a query using the SQL chain:" + "Let's start by defining useful functions to get info from database and running the query:" ] }, { "cell_type": "code", - "execution_count": 41, + "execution_count": 5, "metadata": {}, "outputs": [], "source": [ - "from langchain.chains.sql_database.prompt import PROMPT, SQL_PROMPTS\n", - "from langchain.chat_models import ChatOpenAI\n", - "from langchain.chains import create_sql_query_chain\n", + "def get_schema(_):\n", + " return db.get_table_info()\n", "\n", - "prompt = SQL_PROMPTS['postgresql']\n", + "def run_query(query):\n", + " return db.run(query)" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now let's build the **prompt** we will use. This prompt is an extension from [text-to-postgres-sql](https://smith.langchain.com/hub/jacob/text-to-postgres-sql?organizationId=f9b614b8-5c3a-4e7c-afbc-6d7ad4fd8892) prompt" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "from langchain.prompts import ChatPromptTemplate\n", "\n", - "prompt.template = \"\"\"\n", - "You are a PostgreSQL expert. Given an input question, first create a syntactically correct PostgreSQL query to run, then look at the results of the query and return the answer to the input question.\n", - "Unless the user specifies in the question a specific number of examples to obtain, query for at most {top_k} results using the LIMIT clause as per PostgreSQL. You can order the results to return the most informative data in the database.\n", + "template = \"\"\"You are a Postgres expert. Given an input question, first create a syntactically correct Postgres query to run, then look at the results of the query and return the answer to the input question.\n", + "Unless the user specifies in the question a specific number of examples to obtain, query for at most 5 results using the LIMIT clause as per Postgres. You can order the results to return the most informative data in the database.\n", "Never query for all columns from a table. You must query only the columns that are needed to answer the question. Wrap each column name in double quotes (\") to denote them as delimited identifiers.\n", "Pay attention to use only the column names you can see in the tables below. Be careful to not query for columns that do not exist. Also, pay attention to which column is in which table.\n", - "Pay attention to use CURRENT_DATE function to get the current date, if the question involves \"today\".\n", + "Pay attention to use date('now') function to get the current date, if the question involves \"today\".\n", "\n", - "IMPORTANT NOTE: you can use specialized pgvector syntax (`<->`) to do nearest\n", - "neighbors/semantic search to a given vector from an embeddings column in the table.\n", + "You can use an extra extension which allows you to run semantic similarity using <-> operator on tables containing columns named \"embeddings\".\n", + "<-> operator can ONLY be used on embeddings columns.\n", "The embeddings value for a given row typically represents the semantic meaning of that row.\n", - "The vector represents an embedding representation\n", - "of the question, given below. Do NOT fill in the vector values directly, but rather specify a\n", - "`[embedding]` placeholder.\n", - "The only tables containing 'embeddings' columns are: Track. No other tables contain 'embeddings' column.\n", - "\n", - "FOR EXAMPLE, the query to get 3 songs which title have a certain semantic meaning is:\n", - "'SELECT \"Track\".\"Name\" FROM \"Track\" ORDER BY \"embeddings\" <-> [embedding] LIMIT 3'\n", - "\n", + "The vector represents an embedding representation of the question, given below. \n", + "Do NOT fill in the vector values directly, but rather specify a `[search_word]` placeholder, which should contain the word that would be embedded for filtering.\n", + "For example, if the user asks for songs about 'the feeling of loneliness' the query could be:\n", + "'SELECT \"[whatever_table_name]\".\"SongName\" FROM \"[whatever_table_name]\" ORDER BY \"embeddings\" <-> '[loneliness]' LIMIT 5'\n", "\n", "Use the following format:\n", "\n", - "Question: Question here\n", - "SQLQuery: SQL Query to run\n", - "SQLResult: Result of the SQLQuery\n", - "Answer: Final answer here\n", + "Question: \n", + "SQLQuery: \n", + "SQLResult: \n", + "Answer: \n", "\n", "Only use the following tables:\n", - "{table_info}\n", "\n", - "Question: {input}\n", - "\"\"\"\n", + "{schema}\n", "\n", + "QUESTION: {question}\n", + "SQLQuery:\"\"\"\n", "\n", - "llm = ChatOpenAI(model_name='gpt-4', temperature=0)\n", - "chain = create_sql_query_chain(llm, db, prompt=prompt)" + "\n", + "prompt = ChatPromptTemplate.from_messages([\n", + " (\"system\", template),\n", + " (\"human\", \"{question}\")\n", + "])" ] }, { @@ -279,15 +263,48 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Using the Chain" + "And we can create the chain using **[LangChain Expression Language](https://python.langchain.com/docs/expression_language/)**:" ] }, { - "attachments": {}, - "cell_type": "markdown", + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [], + "source": [ + "from langchain.chat_models import ChatOpenAI\n", + "from langchain.schema.output_parser import StrOutputParser\n", + "from langchain.schema.runnable import RunnablePassthrough\n", + "\n", + "db = SQLDatabase.from_uri(CONNECTION_STRING) # We reconnect to db so the new columns are loaded as well.\n", + "llm = ChatOpenAI(model_name='gpt-4', temperature=0)\n", + "\n", + "sql_query_chain = (\n", + " RunnablePassthrough.assign(schema=get_schema)\n", + " | prompt\n", + " | llm.bind(stop=[\"\\nSQLResult:\"])\n", + " | StrOutputParser()\n", + " )" + ] + }, + { + "cell_type": "code", + "execution_count": 8, "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'SELECT \"Track\".\"Name\" \\nFROM \"Track\" \\nJOIN \"Genre\" ON \"Track\".\"GenreId\" = \"Genre\".\"GenreId\" \\nWHERE \"Genre\".\"Name\" = \\'Rock\\' \\nORDER BY \"Track\".\"embeddings\" <-> \\'[dispair]\\' \\nLIMIT 5'" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "#### Example 1: Getting sad songs" + "sql_query_chain.invoke({\"question\":\"Which are the 5 rock songs with titles about deep feeling of dispair?\"})" ] }, { @@ -295,31 +312,49 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Let's say we want to retrieve songs that express sadness, but filtering based on genre:" + "This chain simply generates the query. Now we will create the full chain that also handles the execution and the final result for the user:" ] }, { "cell_type": "code", - "execution_count": 42, + "execution_count": 10, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "SELECT \"Track\".\"Name\" \n", - "FROM \"Track\" \n", - "JOIN \"Genre\" ON \"Track\".\"GenreId\" = \"Genre\".\"GenreId\" \n", - "WHERE \"Genre\".\"Name\" = 'Rock' \n", - "ORDER BY \"Track\".\"embeddings\" <-> [embedding] \n", - "LIMIT 5\n" - ] - } - ], + "outputs": [], "source": [ - "question = \"Which are the 5 rock songs with saddest titles?\"\n", - "query = chain.invoke({\"question\": question})\n", - "print(query)" + "import re\n", + "from langchain.schema.runnable import RunnableLambda\n", + "\n", + "def replace_brackets(match):\n", + " words_inside_brackets = match.group(1).split(', ')\n", + " embedded_words = [str(embeddings_model.embed_query(word)) for word in words_inside_brackets]\n", + " return \"', '\".join(embedded_words)\n", + "\n", + "def get_query(query):\n", + " sql_query = re.sub(r'\\[([\\w\\s,]+)\\]', replace_brackets, query)\n", + " return sql_query\n", + " \n", + "\n", + "template = \"\"\"Based on the table schema below, question, sql query, and sql response, write a natural language response:\n", + "{schema}\n", + "\n", + "Question: {question}\n", + "SQL Query: {query}\n", + "SQL Response: {response}\"\"\"\n", + "\n", + "prompt = ChatPromptTemplate.from_messages([\n", + " (\"system\", template),\n", + " (\"human\", \"{question}\")\n", + "])\n", + "\n", + "full_chain = (\n", + " RunnablePassthrough.assign(query=sql_query_chain)\n", + " | RunnablePassthrough.assign(\n", + " schema=get_schema,\n", + " response=RunnableLambda(lambda x: db.run(get_query(x[\"query\"]))),\n", + " )\n", + " | prompt \n", + " | llm\n", + ")" ] }, { @@ -327,16 +362,15 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "As we can see, we have a placeholder provided by the LLM for us to insert the actual embedding for \"sadness\", so let's do it:" + "## Using the Chain" ] }, { - "cell_type": "code", - "execution_count": 43, + "attachments": {}, + "cell_type": "markdown", "metadata": {}, - "outputs": [], "source": [ - "final_query = query.replace('[embedding]', \"'\" + str(embeddings_model.embed_query(\"sadness\")) + \"'\")" + "### Example 1: Filtering a column based on semantic meaning" ] }, { @@ -344,27 +378,27 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "And we can run it" + "Let's say we want to retrieve songs that express `deep feeling of dispair`, but filtering based on genre:" ] }, { "cell_type": "code", - "execution_count": 44, + "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "\"[('Sea Of Sorrow',), ('Out Of Tears',), ('My Melancholy Blues',), ('Indifference',), ('Confusion',)]\"" + "AIMessage(content=\"The 5 rock songs with titles that convey a deep feeling of despair are 'Sea Of Sorrow', 'Surrender', 'Indifference', 'Hard Luck Woman', and 'Desire'.\")" ] }, - "execution_count": 44, + "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "db.run(final_query)" + "full_chain.invoke({\"question\":\"Which are the 5 rock songs with titles about deep feeling of dispair?\"})" ] }, { @@ -372,15 +406,13 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "#### Some insights\n", - "\n", - "The previous example couldn't have been used using a standard vector database with metadata filtering on the genre. What is substantially different in implementing this method is that we have combined both in the same solution:\n", - "- Semantic search (songs that have sad titles)\n", + "What is substantially different in implementing this method is that we have combined:\n", + "- Semantic search (songs that have titles with some semantic meaning)\n", "- Traditional tabular querying (running JOIN statements to filter track based on genre)\n", "\n", - "This is something we potentially _could_ achieve using metadata filtering, but it's more complex to do so.\n", + "This is something we _could_ potentially achieve using metadata filtering, but it's more complex to do so (we would need to use a vector database containing the embeddings, and use metadata filtering based on genre).\n", "\n", - "For other use cases, metadata filtering wouldn't be enough:" + "However, for other use cases metadata filtering **wouldn't be enough**." ] }, { @@ -388,40 +420,27 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "#### Example 2: Combining filters" + "### Example 2: Combining filters" ] }, { "cell_type": "code", - "execution_count": 47, - "metadata": {}, - "outputs": [], - "source": [ - "question = \"Get 3 albums which have the most amount of songs in the top 500 saddest songs\"\n", - "query = chain.invoke({\"question\": question})\n", - "\n", - "if '[embedding]' in query:\n", - " final_query = query.replace('[embedding]', \"'\" + str(embeddings_model.embed_query(\"sadness\")) + \"'\")" - ] - }, - { - "cell_type": "code", - "execution_count": 48, + "execution_count": 29, "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "\"[('Unplugged', 9), ('Instant Karma: The Amnesty International Campaign to Save Darfur', 7), ('Lost, Season 2', 7)]\"" + "AIMessage(content=\"The three albums which have the most amount of songs in the top 150 saddest songs are 'International Superhits' with 5 songs, 'Ten' with 4 songs, and 'Album Of The Year' with 3 songs.\")" ] }, - "execution_count": 48, + "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "db.run(final_query)" + "full_chain.invoke({\"question\": \"I want to know the 3 albums which have the most amount of songs in the top 150 saddest songs\"})" ] }, { @@ -429,7 +448,29 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "So we have result for 3 albums with most amount of songs in top 500 saddest ones. This **wouldn't** be possible using only standard metadata filtering, we wouldn't need to run some postprocessing to get the result.\n" + "So we have result for 3 albums with most amount of songs in top 150 saddest ones. This **wouldn't** be possible using only standard metadata filtering. Without this _hybdrid query_, we would need some postprocessing to get the result.\n", + "\n", + "Another similar exmaple:" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "AIMessage(content=\"The 6 albums with the shortest titles that contain songs which are in the 20 saddest song list are 'Ten', 'Core', 'Big Ones', 'One By One', 'Black Album', and 'Miles Ahead'.\")" + ] + }, + "execution_count": 30, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "full_chain.invoke({\"question\": \"I need the 6 albums with shortest title, as long as they contain songs which are in the 20 saddest song list.\"})" ] }, { @@ -437,56 +478,35 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "#### Example 3: Combining filters #2" + "Let's see what the query looks like to double check:" ] }, { "cell_type": "code", - "execution_count": 49, + "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "WITH saddest_songs AS (\n", - " SELECT \"TrackId\" FROM \"Track\" ORDER BY \"embeddings\" <-> [embedding] LIMIT 20\n", + "WITH \"SadSongs\" AS (\n", + " SELECT \"TrackId\" FROM \"Track\" \n", + " ORDER BY \"embeddings\" <-> '[sad]' LIMIT 20\n", "),\n", - "albums_with_sad_songs AS (\n", - " SELECT DISTINCT \"AlbumId\" FROM \"Track\" WHERE \"TrackId\" IN (SELECT \"TrackId\" FROM saddest_songs)\n", + "\"SadAlbums\" AS (\n", + " SELECT DISTINCT \"AlbumId\" FROM \"Track\" \n", + " WHERE \"TrackId\" IN (SELECT \"TrackId\" FROM \"SadSongs\")\n", ")\n", "SELECT \"Album\".\"Title\" FROM \"Album\" \n", - "JOIN albums_with_sad_songs ON \"Album\".\"AlbumId\" = albums_with_sad_songs.\"AlbumId\" \n", - "ORDER BY \"Album\".\"title_len\" ASC \n", + "WHERE \"AlbumId\" IN (SELECT \"AlbumId\" FROM \"SadAlbums\") \n", + "ORDER BY \"title_len\" ASC \n", "LIMIT 6\n" ] } ], "source": [ - "question = \"I need the 6 albums with shortest title, as soon as they contain songs which are in the 20 saddest song list.\"\n", - "query = chain.invoke({\"question\": question})\n", - "print(query)" - ] - }, - { - "cell_type": "code", - "execution_count": 50, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "\"[('Vs.',), ('Faceless',), ('Facelift',), ('Unplugged',), ('By The Way',), ('Black Album',)]\"" - ] - }, - "execution_count": 50, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "final_query = query.replace('[embedding]', \"'\" + str(embeddings_model.embed_query(\"sadness\")) + \"'\")\n", - "db.run(final_query)" + "print(sql_query_chain.invoke({\"question\": \"I need the 6 albums with shortest title, as long as they contain songs which are in the 20 saddest song list.\"}))" ] }, { @@ -494,24 +514,17 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We can even complicate things more, and still use the same solution. This would require a much more complex post-processing if we are limited to just using vectordbs with metadata filtering:" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Combining two semantic similarity filters\n", + "### Example 3: Combining two separate semantic searches\n", "\n", - "One interesting aspect of this approach is that we can even **combine** two semantic search filters:\n", - "- 5 most saddest songs...\n", - "- ...as long as they belong to top 20 albums \"love related\" albums\n", + "One interesting aspect of this approach which is **substantially different from using standar RAG** is that we can even **combine** two semantic search filters:\n", + "- _Get 5 saddest songs..._\n", + "- _**...obtained from albums with \"lovely\" titles**_\n", "\n", - "Let's do the same as we did for the song titles with the album titles:\n", + "This could generalize to **any kind of combined RAG** (paragraphs discussing _X_ topic belonging from books about _Y_, replies to a tweet about _ABC_ topic that express _XYZ_ feeling)\n", + "\n", + "We will combine semantic search on songs and album titles, so we need to do the same for `Album` table:\n", "1. Generate the embeddings\n", - "2. Add them to the table as a new column (which we need to add)\n", - "3. Test the embeddings" + "2. Add them to the table as a new column (which we need to add in the table)" ] }, { @@ -525,29 +538,21 @@ }, { "cell_type": "code", - "execution_count": 61, - "metadata": {}, - "outputs": [], - "source": [ - "albums = db.run('SELECT \"Title\" FROM \"Album\"')\n", - "album_titles = [title[0] for title in eval(albums)]\n", - "album_title_embeddings = embeddings_model.embed_documents(album_titles)" - ] - }, - { - "cell_type": "code", - "execution_count": 58, + "execution_count": 43, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "100%|██████████| 347/347 [00:02<00:00, 162.18it/s]\n" + "100%|██████████| 347/347 [00:01<00:00, 179.64it/s]\n" ] } ], "source": [ + "albums = db.run('SELECT \"Title\" FROM \"Album\"')\n", + "album_titles = [title[0] for title in eval(albums)]\n", + "album_title_embeddings = embeddings_model.embed_documents(album_titles)\n", "for i in tqdm(range(len(album_title_embeddings))):\n", " album_title = album_titles[i].replace(\"'\",\"''\")\n", " album_embedding = album_title_embeddings[i]\n", @@ -557,23 +562,18 @@ }, { "cell_type": "code", - "execution_count": 59, - "metadata": {}, + "execution_count": 45, + "metadata": { + "scrolled": true + }, "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "SELECT \"Album\".\"Title\" FROM \"Album\" WHERE \"Album\".\"embeddings\" IS NOT NULL ORDER BY \"embeddings\" <-> '[-0.005296176500367851, -0.024523600003614955, -0.0007927474517293692, -0.01893319261373208, 0.0035659644242875383, 0.002227528338073427, -0.00032501444871436925, 0.0019508862909545174, -0.017282933652393022, 0.008219307600334083, 0.006239637969211901, 0.009786413422444112, -0.022975684083882866, -0.00395933972790567, 0.012645581285559251, 0.007451745964153147, 0.03661269833460187, -0.01778184876247676, 0.0186133745668314, -0.013918454355509036, -0.0026528854929477575, 0.0038154219793294078, 0.01217225147021595, -0.00718309943805595, -0.006217250836314019, -0.014506918928506913, 0.021440560811520994, -0.026992590259293216, 0.02301406202599352, -0.022898928199661557, 0.018626167214201617, -0.010643523590216577, -0.03175147370746667, -0.017206177768171713, -0.0012264998396480686, -0.00873101535778804, 0.0011041696835090498, 0.0010410057542879437, 0.012747922464520998, -0.01273512981715078, 0.005097890000468161, -0.006901660264588535, 0.002987094802478629, -0.025534222871152652, -0.018843642219495332, 0.013112514777217444, 0.0024386079510046415, 0.004416678734036199, -0.015120967864922619, 0.012025135094135827, 0.02334667272026442, 0.020621827654536573, -0.01829355651993072, -0.035870721061838765, 0.013176478014068536, 0.0290650084464947, -0.0221953288690091, 0.005161853237319253, -0.013496295129646608, -0.01724455571028237, 0.019636790081739314, 0.0026976599915741746, -0.010675506139964732, 0.03326101354773332, 0.0016550551563649536, 0.009025248109948284, -0.022553524858020438, 0.009153174583650468, 0.010886584821573337, 0.010029474653800871, 0.0290650084464947, 0.02704376084877409, 0.0031246162273697825, -0.0004405485109379595, 0.007125532524889967, 0.012524050204219567, -0.022016231805826042, -0.008174533334538318, -0.011993153475710281, 0.00898047384415252, 0.016285103432225546, -0.02213136563215801, -0.02489458863999651, -0.023973514304050345, -0.004730099991590466, 0.007477331258893584, -0.024267747056210588, 0.01347070983490617, -0.016285103432225546, -0.026109895728102923, 0.02816951940528897, 0.009133985612595142, 0.00473329815343302, 0.021913890626864295, -0.00831525338693333, 0.0051298716188937065, 0.005529643246196948, 0.0159908706800653, 0.0060317565181232415, -0.02563656591275962, -0.005369734688407912, 0.00026145074926294394, -0.015722225085290715, -0.0007583670955065809, -0.013368368655944422, -0.018574996624720744, 0.023576940372928352, 0.0019109091515072585, 0.03428442906264124, -0.025252784629007846, -0.013624222534671401, 0.03796872640642591, 0.004554200158927353, -0.027222859774602367, 0.008839753791757507, -0.029218520214937323, -0.011046493845023677, 0.003818620141171962, 0.019214630855876887, 0.01755158110981283, 0.00603815284180835, 0.0076308439586588155, 0.038736285248639014, -0.010413255006230035, -0.003250944085811994, 0.020199668428674145, -0.03689414030203712, -0.0014207888700752186, -0.02154290199048274, 0.010841810555777572, 0.010304517503583178, -0.007733185603281868, 0.011340725665861312, -0.015530334443414827, -0.026787907901369718, 0.01829355651993072, -0.004100059314639378, -0.008053003184521243, -0.01233216049366629, -0.02113353541199053, 0.01029172485621296, 0.04326490104414854, 0.008884528057553272, 0.009767224451388784, -0.021530109343112523, -4.297546710635903e-05, 0.009933528867201624, 0.011334329342176201, 0.001173729936415265, -0.0006612225220853428, 0.0212486692383225, -0.03006283866666218, 0.0036235315702841738, 0.006850489209446356, -0.000268246872282204, 0.004170419340836885, -0.0025089677443714954, 0.00482284668502455, -0.03205849724435191, 0.0075796729035166365, 0.008436783536950408, -0.022758209078589154, 0.004973161222947226, 0.004621362023282304, 0.0059294153391614935, 0.026813493196110157, 0.0158501515589929, 0.017039873352358872, -0.01347070983490617, 0.009536955867402242, -0.024203783819359496, 0.016643299421236882, -0.029192934920196884, -0.0055232469225118385, -0.012581618048708158, 0.016643299421236882, -0.00514586242810648, 0.020046156660231525, -0.027862494005758506, -0.015555919738155264, -0.02352576978344748, 0.014097552350014704, 0.01648978579014904, 0.012696751875040125, -0.009524163220032024, 0.011532616307737198, 0.015555919738155264, -0.0015399208643716827, -0.004762081610016012, -0.009939926122209343, 0.02334667272026442, 0.0020212462007366976, 0.01991823018652934, -0.012396123730517381, -0.6906009578429887, -0.028527715394300306, 0.007905886808441122, -0.012191440441271276, 0.012760715111891217, -0.000906682433031682, 0.002566534890368131, -0.013982417592360127, -0.02671115201714841, 0.003892178096381371, -0.0006228444635593613, 0.025176028744786538, -0.022732621921203497, -0.0017957748595139875, 0.010387669711489599, -0.016732847952828412, 0.0017062259786764797, -0.047384148398520634, 0.0035339825730313396, -0.01297819104850754, -0.008974076589144802, 0.01656654353701557, -0.020954438348807475, -0.02328270948341333, 0.018012118277785913, -0.003927358109480124, 0.005049917107168536, -0.015798980969512023, -0.023717661356645974, -0.0005924618096397662, -0.0029071405235841113, 0.03689414030203712, -0.006722562270082866, -0.014877906633565858, 0.04968683610103127, 0.011404689834035014, -0.02048110853346417, 0.01853661868261009, 0.01591411479584399, 0.05047998396327526, -0.011155232278993144, -0.0005444891491707945, 0.0036747021597650473, 0.011916397591488971, -0.0011161627904186296, 0.014263857697150153, 0.04377661066424772, -0.005248204072729532, 0.0018773283357451087, -0.007522105990350653, 0.025521430223782435, -0.0037738456425455455, 0.0018165630279059187, 0.009760828127703675, -0.006300403509881743, -0.0017158204642041435, 0.03218642744334454, -0.012805489377686982, 0.018306349167300937, 0.02431891764569146, -0.00799543534003265, 0.012037927741506046, -0.020455523238723735, -0.004141635884253893, -0.016604921479126226, 0.016771225894939065, -0.006018963870753023, -0.009875961954035641, 0.016297896079595763, -0.002860766944036417, -0.011136043307937816, 0.01568384714318006, -0.008021021566095698, -0.0064571139058282245, 0.029934912192959993, 0.02132542698518903, 0.0010961742789026632, -0.035973065966090956, -0.01299098369587776, 0.031470035465321865, 0.012927020459026667, -0.004931584653332711, -0.016502578437519257, -0.001415192086850748, 0.03280047451711502, 0.0022579111084083483, -0.014199893528976451, -0.0028527715394300306, 0.0033836685007699683, 0.012959002077452212, 0.014519711575877132, 0.00391776362395246, 0.0007407771471648674, -0.02809276352106766, 0.03661269833460187, 0.010151004803817946, -0.015862944206363118, -0.0002236723169786098, 0.010976134284487475, 0.002971103993265856, -0.0013624223000332706, -0.02228487926324585, 0.02089047511195638, 0.03139327958110055, 0.026173858964954014, -0.01134712198954642, 0.011321536694805984, 0.0015846953629980996, 0.014199893528976451, -0.04303463339148461, -0.014813943396714766, -0.0093066872834157, -0.00955614483845757, -0.007406971698357383, 0.01199954979939539, -0.027888079300498945, 0.006108512868005857, 0.014340613581371463, -0.007752374574337195, -0.01949607096066691, 0.0349496467258926, 0.0004805256794890499, 0.01696311560549234, -0.010278932208842741, 0.01061793829547614, 0.006373961232260499, 0.013982417592360127, -0.023551355078187914, -0.024075855483012092, -0.007490124371925108, 0.00482284668502455, -0.012057116712561374, 0.021376597574669903, 0.002568133971289408, 0.008954887618089474, -0.004016907106732958, 0.0008403202837603583, -0.012434501672628037, 0.00015131361425249615, -0.0015782990393129904, -0.01779464140984698, 0.017078251294469528, -0.0012033130498742214, 0.015069797275441745, -0.004259968338089718, -0.02106957217513944, -0.008788583202276634, -0.018408690346262687, -0.018818056924754894, 0.007387782727302055, -0.0064187359637175685, -0.0055968051105519, -0.01724455571028237, 0.008935698647034146, 0.0007575675550459423, -0.01568384714318006, 0.0015175337314738003, -0.0065626537122938314, -0.018152837398858316, -0.007279044758993894, 0.013649807829411838, 0.01911228967691514, 0.006805714943650592, -0.00371627849654891, 0.002059624142847353, -0.00988235827772075, -0.009613711751623552, 0.03349128120039726, 0.02034038754974655, -0.015952492737954647, -0.00538892365946324, -0.036970896186258435, 0.005491265304086292, 0.02589241886016399, 0.016911945016011468, 0.009652089693734208, -0.01224900735443726, -0.007899490484756012, 0.012357745788406727, -0.001321645503710047, -0.007707599842880126, -0.018869227514235767, -0.022745416431218933, -0.002297888131440281, 0.00279840255527595, 0.004957169948073148, 0.02048110853346417, 0.008545521970919873, 0.0013536272385509193, 0.01577339567477159, -0.017602751699293703, 0.024382880882542552, -0.007534898637720873, -0.011692525331187538, -0.003017477339982898, -0.004541407511557133, -0.0009706459027134267, 0.007368593756246727, 0.03226318332756585, 0.00010613941087938083, 0.029755813267131715, -0.01444295476033321, 0.02564935856012984, -0.01797374033567526, -0.012357745788406727, -0.02914176433071601, 0.01126396978164, -0.012338556817351399, -0.0034508301322942673, -0.024037477540901436, -0.001647059751758567, -0.040194654499424796, -0.018012118277785913, -0.016361859316446858, 0.005475274494873519, 0.018255178577820063, 0.01485232133882542, 0.014020796465793392, -0.02645529720709882, -0.011679732683817319, 0.015645469201069403, 0.00041176493793964176, -0.011116853405559879, 0.004362309517051466, -0.0020884078322609967, 0.01069469511102006, 0.012344953141036508, 0.013496295129646608, 0.008865339086497944, -0.008174533334538318, -0.03674062480830406, 0.0036938913636510276, -0.0033101103127299073, 0.026762322606629283, 0.014110344997384922, -0.016208347548004234, 0.02618665161232423, -0.033772719442542065, 0.017666714936144797, -0.0075796729035166365, 0.012952605753767103, -0.014263857697150153, 0.025457466986931344, 0.02648088250183926, -0.011782073862779067, 0.014020796465793392, 0.022054609747936698, 0.006824903914705919, -0.021376597574669903, 0.0005189037962226946, -0.019073911734804484, -0.00657544635966405, 0.0045062277312890335, 0.003073445405058256, 0.01444295476033321, -0.023576940372928352, -0.007650032929714143, -0.005289780176682742, 0.042932292212522864, 0.0326469627486724, 0.007656429253399252, 0.009268309341305046, 0.01363701518204162, 0.004570190968140125, -0.009920736219831404, -0.012466483291053583, -0.015325651154168724, -0.0034604246178219313, -0.045337317368704805, 0.005600003272394454, -0.009114796641539814, -0.020225253723414584, 0.023065232615474395, -0.021760376995776456, 0.00482284668502455, 0.0008067394679982087, -0.0047652797718585665, -0.008046606860836135, -0.0043814989537680985, 0.005040322621640872, -0.011308744047435764, -0.01911228967691514, 0.009377046843951901, 0.016349066669076637, -0.0039657360515907795, -0.013304405419093331, -0.009978304064319998, 0.020122912544452837, -0.00939623581500723, 2.9208325230053062e-05, 0.00543049976341645, 0.022054609747936698, 0.009811998717184548, 0.0005936611203307242, -0.007387782727302055, -0.009364254196581684, 0.01354746665045009, 0.00779714884013296, 0.016617714126496444, -0.023858380477718377, -0.01608042107430205, -0.0018981163877217137, -0.023640903609779444, -0.01672005530545819, 0.006658599033231774, 0.012300177943918134, 0.006639409596515142, 0.002205141205175545, -0.0007435755387771027, 0.010637127266531467, -0.012229818383381932, -0.020519486475574826, -0.00047293001600915113, -0.0011921194834252802, -0.010400462358859816, -0.015735017732660932, -0.021824342095272765, 0.0069464345303842994, 0.016860774426530594, 0.012920624135341558, -0.01700149541024822, -0.020621827654536573, -0.02760664105835414, 0.015658261848439624, 0.10474660644929693, 0.0028831540769342994, -0.008423990889580188, 0.011737299596983303, 0.012722337169780561, 0.009575333809512898, 0.007023190880266915, -0.049405397858886466, -0.009914339896146296, -0.0018645355719595642, 0.017525995815072394, -0.005385725497620685, 0.0027088535580231158, -0.012703148198725233, 0.01877967898264424, 0.011788470186464176, -0.01118081757373358, 0.02393513636193969, -0.018267971225190284, -0.008475161479061062, -0.01273512981715078, 0.006997605119865173, -0.009920736219831404, 0.032723718632893715, 0.02996049748770043, 0.020455523238723735, 0.012645581285559251, 0.0241526113672334, 0.014954662517787168, -0.011711714302242866, 0.0031198189846059505, 0.01020217632462143, -0.009242724046564609, -0.0024274143845557004, -0.02375603929875663, 0.016950322958122124, -0.005494463465928847, -0.0035659644242875383, 0.026378541322877508, -0.006722562270082866, 0.008782186878591525, 0.03945267722666169, 0.012709544522410344, -0.023730454004016195, 0.02089047511195638, 0.003908169138424796, -0.014430162112962993, 0.024357295587802117, -0.007406971698357383, -0.018229593283079628, 0.03267254804341284, 0.005926217177318939, -0.024549185298355394, -0.0034284427665657326, 0.007144721495945295, 0.02122308394358206, -0.007694807195509907, -0.008929302323349037, -0.006908056588273644, -0.0023042844551253903, -0.017167799826061057, -0.04590019757828486, 0.02270703662646306, -0.006594635330719377, -0.004666136289078068, -0.017039873352358872, -0.03986204566779912, -0.00882056482070218, -0.03571720929339615, 0.0024817831358791287, 0.0017877794549076011, -0.012114684557049966, -0.015210517327836758, -0.00632598880462218, 0.018869227514235767, 0.014788358101974328, -0.015287273212058068, 0.010976134284487475, -0.002341063549145421, 0.008641466826196511, 0.002446603355611028, -0.007899490484756012, -0.004240778901373085, 0.011954775533599625, -0.025201614039526973, 0.01248567226210891, -0.01755158110981283, -0.029423202572860817, 0.0013880075947737074, 0.012799093054001872, 0.01535123644890916, 0.005350545717352584, 0.024037477540901436, -0.009818395040869657, 0.01591411479584399, -0.030088423961402613, 0.020775339422979197, -0.008065795831891463, -0.0012272993801087072, -0.00996551141694978, -0.010432444908607973, -0.006255628778424674, -0.016950322958122124, -0.021760376995776456, 0.014404576818222556, 0.026327370733396634, -0.00029383225433413557, 0.02834861646847203, -0.0010625934631405137, -0.01380332052917707, 0.03625450420823576, -0.040194654499424796, 0.021811549447902548, 0.009594522780568224, -0.01803770357252635, 0.0027632225421771964, -0.0047556852863309025, 0.0182807638725605, 0.0017861803739863237, -0.01803770357252635, -0.0025297557963481, -0.008059399508206352, 7.250860928652549e-05, -0.00035299848125204834, 0.009377046843951901, 0.005395319983148349, 0.005609597757922118, -0.013700978418892711, -0.006888867617218316, 0.006620220625459815, -0.005983784090484922, 0.02980698385661259, -0.007515709666665545, -0.03203291194961148, -0.004221589930317758, -0.01991823018652934, -0.003978528698960998, -0.009978304064319998, 0.011168024926363361, -0.00534734755551003, -0.008878131733868163, -0.018421482993632905, -0.001696631493148816, -0.02141497551678056, 0.026557638386060567, -0.03233993921178716, -0.003700287687336137, 0.008775790554906415, -0.001511137291373365, 0.008283271768507785, -0.013080533158791898, -0.006338781451992398, -0.03175147370746667, 0.016873567073900815, -0.0006192464732788242, -0.024139818719863183, 0.006792922296280373, -0.034565867304786044, 0.0019157063942710905, 0.01755158110981283, 0.018728508393163368, 0.004749288962645794, 0.009639297046363989, 0.0056191922434497815, 0.010649919913901686, -0.005558426702779939, -0.0019668772165826165, -0.011993153475710281, -0.02360252566766879, 0.0342588437679008, -0.004077672181741495, 0.01935534997694929, 0.015274480564687849, -0.025585393460633526, 0.001475957394689938, 0.009313083607100809, -0.0044838401327298456, -0.0019268999607200315, -0.017282933652393022, -0.02301406202599352, -0.03139327958110055, -0.004925188329647601, -0.026506467796579693, -0.0054177071160462316, 0.0020708179421269463, -0.0009146778376380686, 0.014161515586865795, 0.0026704756159124606, 0.012293781620233025, -0.007176703114370841, 0.011890812296748534, -0.007496520695610217, 0.02891149481540686, 0.0029119377663479433, -0.007451745964153147, -0.022898928199661557, 0.00017300123655881428, -0.011532616307737198, 0.0027680197849410284, 0.01428944299189059, 0.016975908252862562, 0.01574781038003115, -0.005772704477553708, 0.007400575374672274, -0.004160824855309221, 0.007074361469747789, -0.03384947532676338, -0.001045003456591137, -0.009133985612595142, -0.010502804469144174, 0.00206122322376863, 0.007029587203952024, -0.013368368655944422, -0.035896306356579206, 0.02824627528951028, -0.014468540055073647, -0.0337983047372825, 0.028655641868002492, -0.007368593756246727, -0.016975908252862562, -0.015223309975206975, -0.01249206858579402, 0.010662713492594515, 0.012204233088641495, 0.020135705191823054, 0.021504524048372085, 0.002301086293282836, -0.03454028201004561, 0.006018963870753023, 0.004455056676146854, 0.01876688633527402, 0.018587789272090965, -0.0015095382104520876, 0.012127477204420185, -0.02222091416374954, 0.009281101988675263, 0.040373751562607856, -0.019956608128639995, -0.02245118367905869, 0.0037770438043881, 0.004947575462545484, 0.018843642219495332, 0.006581842683349159, -0.003671503997922493, -0.015978078032695082, 0.03707323736522018, 0.008232100247704302, -0.002022845281657975, -0.016732847952828412, -0.03781521091269285, -0.01427665034452037, 0.014315028286631027, -0.021837134742642986, 0.025278369923748285, 0.002074016103969501, -0.011129646052930098, -0.002568133971289408, -0.0046085689102507805, -0.005942207986531712, -0.001412793465468832, 0.004710910554873833, 0.033823890032022935, -0.002561737647604299, -0.01642582255329795, -0.013726563713633148, -0.027555470468873265, -0.017078251294469528, -0.004442264028776636, 0.026608810838186663, 0.019189045561136448, -0.040194654499424796, -0.0017941757785927103, -0.004138437722411338, -0.006728958593767976, 0.004842035656079877, -0.0124089163778876, -0.008769394231221306, -0.0019620799738187845, -0.02443405147202343, -0.0063227906427796255, 0.018498240740499435, 0.003994519508173771, -0.008360027652729096, -0.011807659157519504, 0.010515597116514392, -0.008974076589144802, -0.007534898637720873, 0.002563336728525576, -0.018907605456346423, 0.002547345919312803, -0.022118572984787793, 0.008353631329043987, 0.011737299596983303, 0.019444900371186038, -0.0003783840072926518, 0.001478356016071854, 0.00200045791592944, 0.02648088250183926, -0.023474599193966605, 0.01199954979939539, 0.004829243008709658, 0.010240554266732085, -0.00030502587899073984, 0.027222859774602367, -0.020302009607635892, -0.03592189537661008, -0.004106455638324487, -0.020788132070349415, -0.005094691838625606, 0.003047860110317819, 0.012319366914973462, 0.019547241550147785, -0.0020804124276546103, -0.0028191906072525547, -0.01208909833098692, 0.021210291296211842, -0.005743921020970716, -0.017180592473431275, -0.03927358016347863, 0.03387506062150381, 0.008264082797452457, -0.010099834214337073, -0.00825768554244474, -0.021363804927299685, 0.030548961129375697, 0.005750317344655826, -0.002598516508793677, 0.010918567371321493, -0.03356803708461857, -0.02301406202599352, 0.003508397278290903, 0.021440560811520994, -0.011948379209914517, -0.02222091416374954, 0.011481445718256324, 0.008878131733868163, -0.005069106078223864, 0.004576587291825234, 0.007950661074236887, 0.004196004635577321, 0.012140269851790403, 0.00783552724790492, 0.004528614864186915, -0.011654147389076882, -0.0032637369660128653, -0.01551754179604461, 0.0005664766282536838, -0.02350018448870704, 0.0022435191472862004, -0.011820451804889722, 0.0024242162227131455, 0.01028532853252785, 0.003988123184488662, -0.023295502130783546, -0.01779464140984698, -0.013253233898289847, -0.023001269378623304, -0.004707712393031279, 0.003447631970451713, 0.01576060302740137, 0.024523600003614955, 0.004963566271758257, 0.0046085689102507805, 0.027657811647835012, -0.016106006369042487, 0.013496295129646608, 0.0011825249978976164, -0.006233241645526791, 0.01282467928006492, -0.015798980969512023, 0.0023890362096143924, 0.025291162571118502, -0.027222859774602367, -0.0047556852863309025, 0.03940150663718081, 0.018063288867266786, 0.004327129736783365, 0.01779464140984698, -0.0021683621111555145, 0.011487842041941433, 0.011091268110819442, -0.015479163853933954, 0.0016886359721271032, 0.0028175915263312775, 0.011602975868273399, -0.025598186108003743, 0.008308857063248222, 0.014967456096479996, -0.03610099243979314, -0.002793605312512118, -0.009172363554705796, 0.013624222534671401, -0.007937868426866668, 0.004474245647202182, -0.03436118494686255, -0.0028943476433832406, -0.003294119503517134, -0.016349066669076637, 0.014813943396714766, -0.01077784731892648, -0.006297205348039189, -0.002863965105878972, 0.02393513636193969, -0.016208347548004234, 0.00898687016783763, 0.005251402234572087, -0.015312858506798505, -0.0205706570650557, 0.01020217632462143, -0.0323655245065276, 0.013445124540165733, -0.0018101665878054833, 0.007496520695610217, -0.021261461885692716, 0.008123362745057445, -0.00914677825996536, -0.012127477204420185, -0.016221140195374455, 0.0012888642284085358, 0.0302163504351048, -0.02390955106719925, 0.002358653439279471, 0.0059581987957444855, -0.013892869060768599, -0.0063963483651583814, -0.027222859774602367, 0.004960368109915702, -0.011334329342176201, -0.025009722466328477, -0.007035983527637134, 0.00015561117333226046, -0.021504524048372085, -0.014570882165358005, -0.014890699280936077, 0.020967230996177692, 0.01642582255329795, 0.2327247499298811, -0.03257020686445109, -0.01437899152348212, 0.004378300791925544, 0.02776015282679676, 0.01877967898264424, 0.02939761727812038, 0.0010553974825794395, 0.003911367300267351, 0.009901547248776078, -0.024715491576813454, 0.0023090819307198746, -0.004905999358592274, -0.007701203519195017, -0.025930796802274645, -0.02237442779483738, -0.03899214192133382, -0.011404689834035014, -0.03840367641701334, -0.01501862668596087, 0.005395319983148349, 0.003994519508173771, 0.002062822304689907, -0.008929302323349037, 0.011078475463449223, -0.0002692462978580023, -0.0010601947253432713, 0.0020804124276546103, 0.006252430616582119, 0.005587210625024236, -0.0158501515589929, -0.012242611030752151, -0.0016486588326798442, 0.006837696562076137, -0.012703148198725233, -0.02155569463785296, 0.02801600763684635, -0.0002796403529501364, 0.023692074199260317, 0.0006100517579814797, 0.015466371206563736, 0.009658486017419317, -0.003254142247654549, 0.004346318707838693, 0.0018965173068004365, 0.032544621569710656, 0.0070999467644882255, -0.012562428146330221, 0.007355800643215204, -0.0024945760160799994, -0.03914565368977645, -0.01692473766338169, 0.04308580398096548, 0.024945759229477386, -0.009722450185593019, -0.0007915481410384112, 0.030395447498287858, 0.016911945016011468, -0.007221477380166605, 0.00018619371236701517, -0.015786188322141806, 0.012338556817351399, -0.0075988618745719645, 0.036459186566159256, -0.010797036289981807, 0.02498413717158804, -0.0008739010995225079, 0.013764941655743804, 0.029116179035975576, 0.007790752516447851, -0.002067619780284392, -0.005625588567134891, -0.01159018322090318, 0.00640594331634735, -0.02475386951892411, -0.014877906633565858, 0.033951816505725124, 0.003524388087503676, 0.005539237731724611, 0.0047556852863309025, -0.016208347548004234, -0.02188830533212386, 0.0018549410864319004, -0.022796587020699807, -0.005836668180066105, -0.03530784457754915, 0.014097552350014704, 0.022579110152760876, -0.009165967231020688, -0.006620220625459815, 0.013253233898289847, -0.024523600003614955, 0.014315028286631027, -0.005980585928642368, 0.009377046843951901, 0.03658711303986144, 0.01299098369587776, 0.036510357155640126, -0.012031531417820936, 0.02287334290492112, -0.009038040757318504, -0.03062571701359701, -0.012293781620233025, 0.01427665034452037, -0.007496520695610217, 0.0003194176654898987, -0.011973964504654953, 0.012831075603750028, -0.004551001997084797, -0.01879247163001446, 0.0011697322341120717, -0.024382880882542552, 0.004394291601138317, 0.011129646052930098, 0.000911479675795514, -0.0012009144284923054, 0.0059294153391614935, -0.011577390573532962, 0.007816338276849593, -0.016707262658087974, 0.012504861233164239, -0.017602751699293703, 0.0003306112610426714, 0.017948153178289603, -0.010010285682745545, -0.0014503719835341749, -0.01876688633527402, 0.011481445718256324, 0.001914107313349813, -0.040885461182707035, 0.0171294218839504, -0.0022898927268338947, 0.011609372191958508, -0.011174421250048472, 0.011168024926363361, 0.007176703114370841, -0.024242161761470152, -0.014263857697150153, -0.013726563713633148, 0.0178074340572172, -0.028655641868002492, -0.024485222061504303, 0.004100059314639378, 0.011666940036447101, 0.014954662517787168, -0.008942094970719254, 0.03308191648455027, -0.009760828127703675, -0.014187100881606232, -0.001473558773308022, -0.021990646511085607, -0.012140269851790403, -0.010809828937352026, -0.016272310784855328, 0.019009948497953393, -0.014711601286430408, -0.045107049716040876, -0.020302009607635892, 0.009268309341305046, 0.01683518913179016, -0.025495844929041996, -0.011705317978557757, -0.0017797839338858882, -0.013496295129646608, -0.019956608128639995, -0.005337753069982366, -0.16405355062563687, 0.021299839827803372, 0.017999325630415695, -0.014008003818423174, 0.012338556817351399, -0.0017510004773028968, 0.0287323977522238, 0.008801375849646851, 0.00142958393155757, 0.005091493676783051, 0.0007335812248114564, 0.016272310784855328, -0.044851196768636505, -0.02284775761018068, 0.024267747056210588, 0.0256237732653894, -0.013496295129646608, 0.03709882265996062, 0.01721897041554193, 0.004087266667269159, -0.0014615656663984422, -0.011219195515844237, 0.025329540513229158, 0.007931472103181559, 0.010573164029680376, 0.011046493845023677, 0.0033069121508873525, 0.01273512981715078, 0.00024386078636931468, -0.006185269217888472, -0.017014288057618437, -0.004522218540501806, 0.04677010132475015, 0.0314956207600623, -0.026608810838186663, 0.0009962312556615267, -0.013521880424387045, -0.019137874971655575, -0.02287334290492112, 0.017935360530919382, 0.024485222061504303, 0.022054609747936698, 0.0005532841524454829, -0.004905999358592274, -0.019368142624319507, 0.0021635648683916825, 0.016784018542309286, 0.004362309517051466, 0.0038506017595975085, 0.004909197520434829, 0.015312858506798505, -0.009811998717184548, 0.004218391768475204, 0.008699033739362495, 0.018600581919461182, -0.0033037139890447977, 0.0007879502089655372, 0.004778072419228785, -3.832811998900214e-05, -0.014110344997384922, -0.014647638049579316, -0.013931247002879253, -0.005692750431489843, -0.009178759878390905, -0.01324044125091963, -0.004509425893131588, -0.009754431804018566, -0.005641579376347664, -0.031649132528504924, 0.004845233817922431, 0.011027304873968349, 0.014801150749344547, -0.006677788004287102, -0.005843064503751215, 0.028041592931586784, 0.00019099096968276286, -0.01982867979229259, 0.017500410520331956, -0.004423075057721308, -0.013867283766028162, -0.00986956563035053, 0.01281828295637981, -0.007023190880266915, 0.01494186987041695, 0.006754543888508413, 0.009741639156648347, 0.027888079300498945, -0.010713884082075388, -0.014315028286631027, -0.012933416782711775, 0.038557188185455955, -0.02929527609915863, -0.0030846392043378497, 0.005344149393667476, 0.0166560920686071, 0.016950322958122124, 0.0012025135094135828, 0.0060317565181232415, 0.011289555076380438, -0.011027304873968349, 0.010726676729445606, 0.006549861064923613, -0.01111045708187477, 0.00555203037909483, -0.008858942762812835, 0.016272310784855328, 0.025188821392156755, 0.02378162459349707, 0.021210291296211842, -0.010029474653800871, -0.013739356361003367, 0.004036096077788286, 0.031163010065791405, 0.005021133650585545, 0.013892869060768599, 0.020545071770315265, -0.009760828127703675, -0.0067161659463977575, 0.004148032207939002, -0.0007435755387771027, 0.018920399966361863, 0.017193385120801496, -0.016042043132191396, 0.0047141087167163875, -0.007016794556581806, -0.012466483291053583, -0.09251678340930194, -0.03390064591624425, 0.014558089517987788, 0.028655641868002492, 0.0037802419662306547, 0.039017727216074265, -0.003332497678458442, 0.0005768707124496493, -0.03766169914425023, 0.023129195852325486, -0.009524163220032024, -0.011551805278792526, -0.00222592925715215, 0.00979920606981433, 0.02098002364354791, -0.0243445029404319, -0.017692300230885232, -0.0027184480435507798, -0.026250614849175326, 0.012280988972862806, -0.012645581285559251, 0.002030840686264361, -0.0020068544724452016, -0.002026043443500529, -0.007035983527637134, -0.00037018865936344245, -0.0006320391788567057, 0.02456197794572561, 0.009709657538222801, 0.008059399508206352, -0.0006596234411640655, 0.0006788124704270562, 0.009038040757318504, -0.014609260107468661, -0.0028527715394300306, -0.01788418994143851, -0.04121807001433271, -0.01020217632462143, 0.00907002237574405, -0.02694141966981234, 0.000906682433031682, 0.004167221178994329, 0.010074248919596636, -0.034821720252190415, 0.0004945176957578894, -0.01696311560549234, -0.018485448093129218, 0.00391776362395246, 0.0058878387695469785, -0.022489561621169347, -0.015338443801538941, -0.004384697115610653, -0.025265577276378067, -0.015722225085290715, 0.012952605753767103, -0.010822621584722244, -0.007515709666665545, 0.029986082782440866, -0.007074361469747789, 0.015389614391019815, 0.004742892638960684, 0.0010090240194470714, -0.017999325630415695, 0.018664545156312273, 0.002841577740150437, 0.013176478014068536, -0.01485232133882542, -0.023871173125088595, 0.026813493196110157, -0.010355688093064051, -0.0005572818547486761, 0.03060013171885657, -0.023039647320733957, 0.005193834855744799, -0.003242948681205608, 0.013841698471287725, -0.00792507577949645, -0.036024236555571826, 0.024779454813664545, -0.014071967055274267, -0.02000777871812087, -0.01591411479584399, 0.014993041391220433, -0.01574781038003115, 0.023564147725558135, 0.0070999467644882255, 0.0166560920686071, -0.003853799921440063, 0.01559429768026592, -0.0006616222923156621, 0.0009018851320601871, 0.012556031822645112, 0.0025377512009544865, -0.016847981779160377, 0.007087154117118007, 0.009006059138892956, 0.004429471381406418, -0.007240666351221933, 0.010605145648105921, 0.017577166404553268, -0.0031358097938187232, -0.019662375376479753, -0.020711376186128103, 0.03187940018116885, 0.012613599667133706, -0.010048663624856199, -0.013125307424587663, -0.0023490589537518074, -0.0004485439446481776, -0.023449013899226167, -0.008142551716112773, 0.000906682433031682, 0.001858139248274455, 0.0054081126305185676, 0.015543127090785046, 0.004989152032159999, -0.03144445017058143, -0.0050595115926962, 0.024357295587802117, 0.010745865700500934, 0.0017589958819092834, 0.007502917019295326, 0.009255516693934826, -0.006134098162746294, 0.017410860126095207, 0.007266252111623675, -0.03182822959168798, 0.013329990713833768, -0.014570882165358005, 0.009351461549211465, -0.006348375937520062, -0.014404576818222556, 0.01307413683510679, -0.01796094768830504, -0.011718110625927975, 0.02156848728522318, -0.0032669351278554197, -0.01322764860354941, 0.009038040757318504, 0.025905211507534207, 0.02295009878914243, 0.020800924717719632, -0.005900631416917198, -0.011858830678322987, 0.011973964504654953, -0.0016982305740700933, 0.02277100172595937, -0.006361168584890281, -0.047332977809039764, 0.013445124540165733, 0.017103836589209966, -0.020621827654536573, 0.01363701518204162, 0.02195226856897495, -0.008673448444622058, -0.01044523755597819, -0.03443794083108386, -0.024766662166294327, 0.0003306112610426714, 0.024421258824653208, -0.0010034272362226008, -0.012242611030752151, 0.006773733325225045, -0.013483502482276389, 0.010797036289981807, 0.004036096077788286, 0.03036986220354742, 0.01762833699403414, 0.026225029554434887, -0.011500634689311652, -0.016464200495408605, -0.001478356016071854, -0.022681451331722623, -0.002018048038894143, 0.009978304064319998, 0.03876187054337945, -0.0037802419662306547, 0.002139578654572523, -0.016131591663782925, -0.007573276579831528, -0.023244331541302673, 0.01217225147021595, 0.015287273212058068, -0.015210517327836758, -0.013944039650249472, 0.017525995815072394, 0.0045989744247231165, 0.033772719442542065, 0.0007607657168884969, 0.011052890168708786, 0.009121192965224923, 0.017321311594503678, -0.009114796641539814, -0.003661909512394829, -0.003671503997922493, 0.01494186987041695, -0.0031310125510548917, 0.01192279391517408, -0.004467849323517073, -0.010905774723951274, -0.006377159394103054, 0.020519486475574826, 0.002369847238559065, -0.00656905003597894, -0.011411086157720123, -0.018754093687903803, -0.009978304064319998, 0.01020857264830654, -0.02996049748770043, 0.017180592473431275, 0.00882056482070218, 0.01322764860354941, 0.011622165770651336, 0.018830849572125115, 0.017858604646698074, 0.010579560353365484, -0.018395897698892466, 0.009287498312360372, 0.01412313764475514, -0.011129646052930098, -0.018626167214201617, 0.03077922878203963, 0.017014288057618437, 0.024869003345256074, 0.014545296870617569, -0.01920183820850667, 0.0028783568341704674, 0.0013184473418674924, 0.014340613581371463, -0.020532279122945047, 0.0027376372474367597, 0.017346896889244116, 0.0028911494815406858, -0.008059399508206352, -0.02049390118083439, -0.013892869060768599, 0.003169390725996199, -0.011852434354637878, 0.012645581285559251, 0.03428442906264124, -0.003920961785795015, 0.03525667398806828, 0.026992590259293216, -0.015120967864922619, 0.00694003820669919, 0.010342895445693834, 0.004953971786230593, -0.009332272578156137, 0.030983913002608346, -0.0005852658872863552, -0.015466371206563736, 0.01574781038003115, -0.009351461549211465, 0.0202380463707848, -0.03454028201004561, -0.007643636606029034, -0.0031661925641536448, -0.0012496866294219157, 0.002513764987135327, -0.009933528867201624, 0.00019828680548992212, 0.005184240370217135, -0.006901660264588535, 0.014622052754838879, 0.01550474914867439, -0.008142551716112773, -0.004138437722411338, -0.00518743853205969, -0.009293894636045482, -0.00906362605205894, -0.019470485665926476, -0.019623997434369097, 0.0016038844504687535, -0.003052657353081651, -0.015325651154168724, 0.02278379437332959, 0.0012632788172527729, 0.008590296236715637, -0.028118348815808096, -0.005683155480300874, 0.013432331892795515, -0.010087041566966855, -0.0063323851283072895, -0.0070807577934328975, -0.028297445878991155, 0.01028532853252785, 0.00831525338693333, -0.013777735234436633, 0.011570994249847854, -0.009121192965224923]' LIMIT 5\n" - ] - }, { "data": { "text/plain": [ "\"[('Realize',), ('Morning Dance',), ('Into The Light',), ('New Adventures In Hi-Fi',), ('Miles Ahead',)]\"" ] }, - "execution_count": 59, + "execution_count": 45, "metadata": {}, "output_type": "execute_result" } @@ -581,7 +581,6 @@ "source": [ "embeded_title = embeddings_model.embed_query(\"hope about the future\")\n", "query = 'SELECT \"Album\".\"Title\" FROM \"Album\" WHERE \"Album\".\"embeddings\" IS NOT NULL ORDER BY \"embeddings\" <-> ' + f\"'{embeded_title}' LIMIT 5\"\n", - "print(query)\n", "db.run(query)" ] }, @@ -590,167 +589,36 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "#### Combining both filters" + "Now we can combine both filters:" ] }, { "cell_type": "code", - "execution_count": 104, + "execution_count": 54, "metadata": {}, "outputs": [], "source": [ - "from langchain.chains.sql_database.prompt import PROMPT, SQL_PROMPTS\n", - "from langchain.chat_models import ChatOpenAI\n", - "from langchain.chains import create_sql_query_chain\n", - "\n", - "prompt = SQL_PROMPTS['postgresql']\n", - "\n", - "prompt.template = \"\"\"\n", - "You are a PostgreSQL expert. Given an input question, first create a syntactically correct PostgreSQL query to run, then look at the results of the query and return the answer to the input question.\n", - "Unless the user specifies in the question a specific number of examples to obtain, query for at most {top_k} results using the LIMIT clause as per PostgreSQL. You can order the results to return the most informative data in the database.\n", - "Never query for all columns from a table. You must query only the columns that are needed to answer the question. Wrap each column name in double quotes (\") to denote them as delimited identifiers.\n", - "Pay attention to use only the column names you can see in the tables below. Be careful to not query for columns that do not exist. Also, pay attention to which column is in which table.\n", - "Pay attention to use CURRENT_DATE function to get the current date, if the question involves \"today\".\n", - "\n", - "IMPORTANT NOTE: you can use specialized pgvector syntax (`<->`) to do nearest\n", - "neighbors/semantic search to a given vector from an embeddings column in the table.\n", - "The embeddings value for a given row typically represents the semantic meaning of that row.\n", - "The vector represents an embeddings representation\n", - "of the question, given below. Do NOT fill in the vector values directly, but rather specify a\n", - "`[search_word]` placeholder, which should contain the word that should be embedded for filtering.\n", - "The only tables containing 'embeddings' columns are: 'Track' and 'Album'. No other tables contain 'embeddings' column.\n", - "The column is ALWAYS called 'embeddings'. MAKE SURE you call it 'embeddings' and not 'embedding' or similar.\n", - "FOR EXAMPLE, the query to get 3 songs which title are saddest:\n", - "'SELECT \"Track\".\"Name\" FROM \"Track\" ORDER BY \"embeddings\" <-> [sad] LIMIT 3'\n", - "\n", - "Or three happiest albums:\n", - "'SELECT \"Album\".\"Title\" FROM \"Album\" ORDER BY \"embeddings\" <-> [happiness] LIMIT 8'\n", - "\n", - "Use the following format:\n", - "\n", - "Question: Question here\n", - "SQLQuery: SQL Query to run\n", - "SQLResult: Result of the SQLQuery\n", - "Answer: Final answer here\n", - "\n", - "Only use the following tables:\n", - "{table_info}\n", - "\n", - "Question: {input}\n", - "\"\"\"\n", - "\n", - "\n", - "llm = ChatOpenAI(model_name='gpt-4', temperature=0)\n", - "chain = create_sql_query_chain(llm, db, prompt=prompt)" + "db = SQLDatabase.from_uri(CONNECTION_STRING) # We reconnect to dbso the new columns are loaded as well." ] }, { "cell_type": "code", - "execution_count": 113, - "metadata": {}, - "outputs": [], - "source": [ - "# question = \"I want to know 5 songs that have a happy name but belonging to an album with a sad name\"\n", - "question = \"I want the 3 saddest songs and album title for each one of the top 5 happiest albums\"\n", - "query = chain.invoke({\"question\": question})" - ] - }, - { - "cell_type": "code", - "execution_count": 114, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "SELECT \"Track\".\"Name\", \"Album\".\"Title\" \n", - "FROM \"Track\" \n", - "JOIN \"Album\" ON \"Track\".\"AlbumId\" = \"Album\".\"AlbumId\" \n", - "ORDER BY \"Album\".\"embeddings\" <-> [happiness] DESC, \"Track\".\"embeddings\" <-> [sad] ASC \n", - "LIMIT 15\n" - ] - } - ], - "source": [ - "print(query)" - ] - }, - { - "cell_type": "code", - "execution_count": 115, - "metadata": {}, - "outputs": [], - "source": [ - "final_query = query.replace('[happiness]', \"'\" + str(embeddings_model.embed_query(\"happiness\")) + \"'\")\n", - "final_query = final_query.replace('[sad]', \"'\" + str(embeddings_model.embed_query(\"sad\")) + \"'\")" - ] - }, - { - "cell_type": "code", - "execution_count": 117, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[('Leila', 'A TempestadeTempestade Ou O Livro Dos Dias'), ('Aloha', 'A TempestadeTempestade Ou O Livro Dos Dias'), ('Soul Parsifal', 'A TempestadeTempestade Ou O Livro Dos Dias'), ('Nat�lia', 'A TempestadeTempestade Ou O Livro Dos Dias'), ('Mil Peda�os', 'A TempestadeTempestade Ou O Livro Dos Dias'), ('Dezesseis', 'A TempestadeTempestade Ou O Livro Dos Dias'), ('M�sica De Trabalho', 'A TempestadeTempestade Ou O Livro Dos Dias'), ('M�sica Ambiente', 'A TempestadeTempestade Ou O Livro Dos Dias'), ('Longe Do Meu Lado', 'A TempestadeTempestade Ou O Livro Dos Dias'), ('Esperando Por Mim', 'A TempestadeTempestade Ou O Livro Dos Dias'), ('Quando Voc� Voltar', 'A TempestadeTempestade Ou O Livro Dos Dias'), (\"L'Avventura\", 'A TempestadeTempestade Ou O Livro Dos Dias'), ('A Via L�ctea', 'A TempestadeTempestade Ou O Livro Dos Dias'), ('1� De Julho', 'A TempestadeTempestade Ou O Livro Dos Dias'), ('O Livro Dos Dias', 'A TempestadeTempestade Ou O Livro Dos Dias')]\n" - ] - } - ], - "source": [ - "print(db.run(final_query))" - ] - }, - { - "cell_type": "code", - "execution_count": 120, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "SELECT \"Track\".\"Name\" \n", - "FROM \"Track\" \n", - "ORDER BY \"embeddings\" <-> [happiness, hope] \n", - "LIMIT 3\n" - ] - } - ], - "source": [ - "question = \"Three songs about a combination of happiness with hope\"\n", - "query = chain.invoke({\"question\": question})\n", - "print(query)" - ] - }, - { - "cell_type": "code", - "execution_count": 126, + "execution_count": 49, "metadata": {}, "outputs": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "SELECT \"Track\".\"Name\" \n", - "FROM \"Track\" \n", - "ORDER BY \"embeddings\" <-> '[0.002974223739582622, -0.02512427741888291, 0.016198377071763158, -0.032035069414131874, -0.0022589244241709074, 0.013963673089441988, -0.015294162454309174, -0.024465491458854405, -0.009106743643139601, -0.01993149627016118, 0.014377029460722387, 0.01652131132937232, -0.0010527653491863428, 0.0001054581864862774, 0.009868868230326892, 0.022747482474972422, 0.041335573798101495, -0.017554701326250696, 0.028030685737365387, -0.023277093986899162, -0.010527654190355397, 0.011489997309455175, -0.018768933687826294, -0.03309429243798535, -0.017296354292692415, -0.007401651172745169, 0.01659881599823338, -0.020409439865492467, -0.008002308631127879, -0.019440637092664973, 0.030071628978856534, -0.002683582861168243, -0.03673698952271214, 0.003384350051168509, -0.02354835846526762, -0.020448192199922995, -0.0018342661035888998, -0.011515832199075529, 0.002099071975967597, 0.012478176249497933, -0.012775275617486743, -0.0037234308819597372, -0.00331330457037385, 0.006639526585557877, -0.026609774258826966, 0.011787096677443987, -0.002328355224364294, -0.008383370459060213, -0.011864601346305046, -0.0035038357171706734, 0.026687278927688025, 0.04252397127005674, -0.03162171490549672, -0.010760166334293325, 0.011373741237486211, 0.010224096099961498, -0.007207890431915146, -0.0015081026011925197, -0.00946842930385667, -0.007750419854313374, -0.0008347848067908868, 0.0039075033063514725, -0.030846671942176628, 0.028599049583722655, -0.004592123690339018, -0.01510040171347915, 0.004382216609157586, 0.007711667985544157, -0.002036099898179299, -0.010618075372704008, 0.01693466770065272, 0.016456725967966688, -0.020486942671708277, -0.01519082289582776, 0.011987316140679098, 0.00597105254192174, -0.015242492675068466, 0.006539416853940321, 0.0048892230583278285, 0.02354835846526762, 0.01825223962070973, -0.02896073617576235, 0.0010681046984830993, -0.00042950244969731986, -0.0115868781455315, 0.0031017825757604905, 0.003368203245155788, 0.020189843303719465, -0.0039010448167770403, -0.019014363276574398, 0.010204719932746234, 0.007963556296697349, 0.00903569769668363, 0.010508278023140132, -0.0036136332995652053, 0.02152033266858665, 0.01893685860771334, 0.03198339963489116, -0.01436411201591221, -0.02748815678062847, -0.020189843303719465, 0.009106743643139601, -0.012568597431846544, -0.006943084675951777, 0.006697654621542359, -0.026377262114889038, 6.312353413020557e-05, -0.02208869651494392, 0.007692291818328892, -0.0058838602551143605, -0.01862684179491435, 0.0054834222599667625, -0.005560926463166509, -0.034463534137283064, 0.0024284649559818496, -0.01956980967812149, 0.005922612589544891, 0.0054898809823718505, -0.00742748559670421, -0.026842288265410144, 0.0018213486587787232, 0.03335264133418888, 0.024310483983777536, -0.007711667985544157, -0.016082121931116817, 0.024685088020627407, -0.027255642774045293, -0.03244842392276702, -0.011502914754265353, -0.00820898681676808, 0.03593611539506219, 0.032164243862233634, 0.01378282979342214, -0.01030805855990502, -0.02246330055179379, 0.03735702314831011, -0.016004617262255758, -0.012400671580636874, -0.007847300224728385, -0.019957331159781537, -0.005144341196344878, 0.024969269943806043, -0.017490114102199812, -0.02110697629730625, 0.0007810970260445748, 0.010760166334293325, 0.01145770462875236, 0.011425411016726918, 0.00813148214790702, 0.0009074450263944815, 0.006132519670726322, -0.030898341721417335, 0.02587348549258265, 0.014467450643070997, 0.007950638851887173, 0.0017745232705878176, -0.008822561719960965, 0.021416993110105236, 0.023160838846252824, -0.012316709120693351, -0.0036136332995652053, -0.03658198391028052, -0.01124456772070707, -0.012568597431846544, 0.006833286860726588, 0.034876888645918216, 0.02080987692931744, 0.009784904839060745, -0.003432790236376015, -0.0022912178033656925, 0.006206794512723525, 0.027023130630107365, -0.03495439331477927, 0.01836849476135607, -0.004546913099164712, 0.010172426320720792, 0.006090538440754561, 0.013653656276643, -0.008519003629567068, -0.007731043687098109, -0.021701175033283872, 0.027436487001387764, 0.03534191665908457, 0.016004617262255758, 0.015875442814153994, 0.011334988903055681, 0.005638430666366256, -0.018110147727797787, 0.0019569810143782802, 0.013498647870243506, -0.014377029460722387, -0.003167984247581989, 0.01896269349733369, -0.011832307734279604, -0.676663316215577, -0.029968291283020372, 0.003067874515964433, -0.004411280859980483, 0.020499860116518453, 0.02114572863173678, 0.017141344954970297, -0.006164813282751764, -0.021016556046280263, 0.0004621995281462551, -0.008745057051099906, 0.009403843011128411, -0.008137940870312109, -0.00963635515506634, 0.008441498960706008, -0.005199239871126816, -0.001273167970691371, -0.00842858151589583, -0.0265064365629908, 0.005179864169572864, -0.005428523119523512, 0.020293182862200877, -0.005557697101963965, -0.0037557242611545227, 0.019091867945435458, 0.010495360578329956, 0.00378478851197742, -0.005919383228342346, -0.009494264193477021, 0.005996887431542093, 0.0003931723398955225, 0.031492540457394955, 0.018265157065519906, -0.0018116606915864189, 0.04327317934375648, -0.0047568201803461435, -0.02124906819021819, 0.03787371721542668, 0.0252405344221745, 0.05244450927962434, -0.023031664398151058, -0.005977511264326828, -0.010379104506360991, -0.008441498960706008, -0.011955022528653656, 0.008370453945572662, 0.031208358534216322, 0.024530078682905288, 0.018303907537305188, -0.031492540457394955, 0.022902491812694544, -0.013065916263070467, 0.0153199964126069, 0.008906524179904488, -0.008228362983983344, -0.015978782372635406, 0.03270677281897055, -0.00993345452305515, 0.02246330055179379, 0.02666144403806767, -0.018536421543888366, -0.001756761900389153, -0.030329976012414814, -0.007123927506310311, -0.022334126103692023, 0.01862684179491435, 0.016650485777474088, -0.014454533198260821, 0.008609424811915678, -0.007227266599130411, -0.0035490465411756353, 0.00826065566468616, 0.004669627893538764, -0.0031066266175643065, 0.007937721407076997, 0.031234191561191425, 0.01510040171347915, -0.012516928583928463, -0.006878497917562206, 0.01783888324942933, -0.000703996492995146, 0.0006846204421952093, -0.0032438737701804637, 0.0075824940031037034, 0.026971460850866658, -0.017748461135758096, -0.023186673735873176, 0.003001673076973591, -0.005228304121949713, 0.0057353105711199556, 0.017554701326250696, 0.01175480399674117, -0.01852350409907819, -0.011444787183942182, -0.0024155475111716732, -0.0021216775043854065, -0.024814260606083924, -0.016986337479893428, -0.016611733443043557, 0.0024462262097651863, -0.013705326055883706, -0.011657923160664847, 0.011858142623899958, 0.018562256433508718, 0.01765804088473211, -0.005790209711563206, -0.011535208366290794, 0.017257601958261884, 0.04366069896277128, -0.025111359974072735, 0.0040592823515484214, -0.006345656113110298, -0.020719456678291455, -0.0089000654574994, -0.009261752049539095, -0.030743332383695216, 0.020112340497503656, 0.013976590534252164, 0.013136962209526438, -0.01569460044945677, 0.013149879654336614, -0.016107956820737172, 0.02738481722214706, -0.004327317934375648, 0.01301424648382976, 0.009681566211901957, -0.005315496408757093, -0.01893685860771334, -0.0022766859107849003, -0.0035587343919526115, 0.014260772457430798, 0.03208673919337258, 0.00907445003111416, -0.01104434825747196, 0.00933279706467244, -0.01373116094550406, -0.003461854254368256, -0.01998316604940189, 0.031105018975734908, -0.006203565151520981, -0.013924920755011458, -0.0007601062946433661, -0.004191685229530106, -0.0023654926453628954, -0.013317804574223659, -0.023264176542088986, -0.007860217669538563, -0.01198085741827401, -0.022062861625323567, -0.009558851417527905, -5.7825399787071635e-05, -0.021158646076546956, -0.04332484912299719, 0.031053349196494204, 0.004014071760374116, -0.014428698308640468, -0.007278935912709805, 0.0010164352684883776, 0.01001095919191621, -0.013744078390314236, 0.019957331159781537, 0.015113319158289326, -0.03004579408923618, -0.00505069018713241, -0.006500664519509792, -0.021003638601470087, -0.004314400489565471, 0.032035069414131874, 0.002244392531590115, -0.0314150357885339, 0.0030210490113581996, 0.02152033266858665, -0.02106822582552097, 0.004653481087526044, -0.002359034155788463, 0.013304887129413483, -0.024310483983777536, -0.017154262399780473, 0.016456725967966688, -0.003765412344762155, -0.01975065390546396, -0.02494343505418569, -0.027617329366084988, -0.008783809385530436, 0.018536421543888366, -0.015203740340637937, 0.022024109290893035, 0.030019959199615826, 0.017980975142341274, -0.010120756541480085, -0.027953181068504328, 0.039191289135483684, -0.011315612735840417, 0.014519120422311705, -0.006135749031928866, -0.004240125647568269, -0.01927271031013268, 0.01854933898869854, 0.029735777276437194, 0.007039964580705476, 0.016870080476601837, -0.014105764051031304, 0.014105764051031304, -0.004572747988785065, 0.019027280721384574, -0.04037968660743893, -0.014699962787008925, -0.026364344670078862, 0.0072466427663456755, 0.006736406490311576, -0.017244684513451708, -0.018807686022256822, -0.0089000654574994, -0.02648060167337045, 0.02016400841409911, 0.03126002645081178, 0.0026141520609748565, 0.00759541144791388, -0.025576386124593842, 0.002328355224364294, 0.02077112459488691, -0.009604062474363522, 0.006155125199144131, -0.010966844519933525, 0.025847650602962298, 0.018123065172607963, -0.006794534991957371, 0.019053115611004926, -0.008622342256725854, -0.010456608243899426, -0.018794768577446646, 0.018562256433508718, 0.011935646361438393, 0.001327259421588001, -0.014338277126291857, -0.006587856806317171, 0.03244842392276702, -0.02629975930867323, 0.009281128216754359, 0.011076641869497401, 0.007950638851887173, -0.003050113262181097, -0.0009373163846873586, -0.01663756833266391, -0.006781617547147194, -0.0021071453789739576, 0.013756995835124412, -0.0022363191285837544, 0.012445882637472492, 0.0054834222599667625, 0.0046922334219565735, -0.006335968495163978, -0.02025443052777035, -0.00025330147682145834, 0.0023800247707743438, -0.010372645783955903, -0.011406034849511653, 0.0021523562029789195, 0.010256388780664314, 0.03769287671337471, 0.022734565030162246, -0.0012505625586888903, 0.014816219790300515, -0.015332913857417078, -0.006051786106324031, 0.0039236501123641925, -0.004718068311576926, -0.010721413999862796, -0.016844245586981485, 0.007227266599130411, 0.013589069983914742, -0.0027417111299833812, -0.008241279497470895, -0.029529100022119618, 0.01941480220304462, -0.006549104937547954, 0.0031615255251769008, 0.004039906184333157, 0.008531921074377244, 0.014015342868682694, -0.009668648767091781, -0.006361802919123019, 0.012181076881509122, 0.021210315855787663, 0.002815985971980584, -0.0140670126479234, -0.023083334177391765, -0.003361744755581356, -0.005270285351921474, 0.02197244137429758, 0.010721413999862796, 0.017942222807910743, 0.021429910554915416, -0.00993345452305515, -0.0064651420119431185, -0.017851800694239507, 0.01938896731342427, 0.010030335359131474, 0.017929305363100567, 0.006084079718349473, 0.010327434727120286, -0.003003287757574863, 0.005363936361133942, -0.022863739478264013, 0.00893881779192993, 0.00162435890599214, -0.016792575807740778, -0.03231924947466525, -0.024853012940514452, -0.009688024934307045, -0.010798918668723855, 0.005974281903124284, -0.024052136950219256, 0.0023364286273706543, -0.018575173878318894, -0.009423219178343677, -0.024995104833426394, -0.0005316303865473056, 0.02474967524467829, -0.0140670126479234, 0.007337064414355599, -0.011360823792676035, -0.003539358457568003, 0.019634396902172373, 0.07972598508064473, 0.024736757799868114, 0.024362153763018243, -0.004201373313137739, 0.007182056007956105, -0.0002837783985474765, -0.02307041673258159, -0.02790151128926362, -0.009823657173491275, 0.012064820809540158, -0.0024801345023918995, -0.014880806083028773, 0.010979761964743701, -0.0005211350208467013, 0.011645005715854669, -0.003678220057954776, 0.015759187673507653, -0.004114181491991672, 0.004236896286365724, -0.006678278454327094, 0.0009405457458899027, -0.0178001309149988, 0.021662424561498594, 0.01927271031013268, 0.0075760352806986155, 0.0010753707611888237, 0.01526832756468882, 0.04867263774679578, 0.0024833636307637877, -0.019918578825351006, -0.016095039375926996, 0.02456883101733582, 0.002730408365774477, 0.004808489493925538, -0.012723605838246037, 0.010534112912760485, 0.015591261822297984, 0.01241358902544705, 0.03071749749407486, -0.01232316784309844, -0.001869788843986229, 0.019608562012552018, 0.014351194571102035, -0.02414255906389049, 0.023574193354887973, -0.014415780863830292, -0.027178139967829484, 0.008964652681550283, -0.003218038880560111, -0.0013934609769941716, 0.03309429243798535, -0.014919558417459303, -0.012135865824673504, -0.02287665692307419, 0.01656006366380285, 0.013182172335039431, -0.0025059691591815965, -0.0066007742511273474, 0.004802031237181761, -0.02024151308296017, -0.015177905451017585, -0.028340702550164375, 0.023341681210950045, -0.005964593819516652, -0.007156221118335753, -0.03490272353553857, -0.009481346748666845, -0.025137194863693087, -0.029089908761218863, 0.005231533483152257, -0.001407185645689656, -0.005434981841928601, -0.01982815857432502, 0.0019440636859834317, 0.02388421016768696, 0.0021846495821737045, -0.011315612735840417, 0.012458800082282668, 0.002901563578186691, 0.02535679142546609, -0.006639526585557877, -0.005351018916323765, -0.025137194863693087, -0.004653481087526044, -0.023367516100570397, 0.0010995908537925767, 0.025602221014214194, -0.01938896731342427, -0.03242258903314667, 0.004737444013130879, 0.013705326055883706, 0.007117468783905223, 0.010921633463097907, -0.014389945974209938, -0.003335909865961003, -0.010508278023140132, 0.008292949276711602, -0.0025850880429826154, -0.017244684513451708, -0.00953947525031264, 0.025589303569404018, -0.014699962787008925, 0.0012214984242813211, -0.009358631954292793, 0.013705326055883706, 0.006281069354720728, 0.011761262719146258, 0.028340702550164375, -0.020267347972580525, -0.032500093702007724, 0.02880572683804023, -0.03195756474527081, 0.001463699117488194, 0.00637794972513574, 0.02073237412310163, 0.007356440115909552, -0.005031314485578458, 0.020138175387124008, -0.0031857455013653254, -0.013769912348611964, 0.003600715854755029, -0.03629780012445664, 0.005974281903124284, 0.018135982617418143, -0.007776254743933727, -0.010107839096669909, 0.030820837052556276, -0.005990428709137005, -0.01004971152634674, 0.022592474999895557, 0.0018827061723810776, 0.013847417017473024, -0.009998041747106032, -0.03175088562830799, -0.002496281075573964, -0.0004795572281946017, -0.004359611080739777, 0.017322189182312767, -0.01783888324942933, -0.021429910554915416, 0.0030242783725607436, -0.0032955430837598577, 0.01074079016707806, -0.023470855659051812, 0.02681645337578979, -0.03547108738189583, -0.0173867745437184, -0.001294966042393216, -0.019582727122931666, 0.033636821394722266, -0.003600715854755029, 0.00401730112157666, -0.03172505073868764, -0.0032148095193575664, 0.004253043092378445, -0.03529024687984386, -0.007802089633554081, -0.01994441371497136, 0.0011230036060956934, 0.00933279706467244, 0.012639643378302515, 0.023457938214241636, 0.022321208658881847, -0.009894703119947246, 0.012329626565503527, -0.004256272453580989, -0.018303907537305188, -0.004420968943588116, -0.009177788658272946, 0.03164754979511707, 0.018239322175899554, 0.011012054645446518, 0.016534228774182497, -0.008460875127921272, 0.008984028848765547, 0.03831291033897268, 0.019608562012552018, -0.02314792140144265, -0.01994441371497136, -0.018975610942143867, -0.009378008121508059, -0.0057449986547275875, -0.03647864435179911, 0.0025544093443891024, -0.007259560211155852, -0.003251946940356168, 0.029761612166057546, -0.00027550321867362117, -0.005283202796731651, 0.016430891078346336, 0.039087949577002276, -0.029089908761218863, 0.01604336959668629, 0.008273573109496337, 0.004475867618370054, -0.0157333527838873, -0.012129407102268416, 0.011967939973463834, -0.00570624678595837, 0.014609541604660315, 0.018303907537305188, 0.015978782372635406, 0.0001706505228609048, -0.005292890880339283, -0.0065975448899248034, 0.005124965029129613, -0.008654635868751296, -0.046605857753039034, 0.008790268107935525, -0.013679491166263352, 0.0027255643239706604, -0.0022783005913861723, -0.004802031237181761, -0.035548592050756896, 0.019776488795084316, 0.00226861250777854, -0.0073822750055299044, 0.03438602946842201, -0.010979761964743701, -0.004585664967933929, -0.0302008034269583, 0.005373624444741574, 0.03539358271303478, -0.008512544907161978, 0.032913451935933376, 0.0025075838397828685, -0.013563235094294388, -0.03299095287950394, 0.009778446116655657, -0.008551296310269883, -0.008493168739946714, 0.01746427921257946, 0.009985124302295856, -0.003626550744375382, -0.018575173878318894, 0.004482326340775142, 0.004882764801584052, -0.024969269943806043, -0.019892743935730654, 0.012174618159104034, 0.018575173878318894, 0.005279973435529107, -0.0007060148437467361, -0.002210484471794058, 0.004714838950374382, 0.02051277756132863, -0.01074079016707806, 0.005018397040768281, -0.005405918056767016, -0.022334126103692023, -0.039372133362826155, 0.023225424207658458, -0.029916621503779665, 0.019066033055815102, 0.010127215263885174, -0.009926995800650061, -0.01588836025896417, 0.010114297819074997, -0.018497669209457834, -0.0038396871867593576, -0.0044500327287497, 0.03219007875185399, -0.011444787183942182, -0.02061611711981004, 0.008221904261578256, 0.0017422297749777043, -0.008990487571170637, 0.017141344954970297, -0.0009098669890887256, 0.020603199674999864, -0.03283594726707232, -0.025899320382203005, -0.015746270228697477, -0.00502485576317337, 0.022101613959754095, -0.0032858552329828815, -0.005625513221556079, -0.010495360578329956, -0.03299095287950394, -0.007647081227154587, 0.006956002120761953, 0.007802089633554081, -0.00238486881257816, -0.007892511281564004, 0.006151895837941587, -0.01649547643975197, -0.01570751789426695, -0.011393117404701477, 0.02012525794231383, 0.0019505224083885202, -0.0031518374415692683, -0.009830115895896363, -0.00759541144791388, 0.017670958329542286, -0.008144399592717197, -0.017528866436630344, -0.002166888328390368, 0.01543625341589849, -0.015345831302227254, 0.011858142623899958, -0.026532271452611157, 0.0009050230054925734, 0.0018746327693747171, -0.00378478851197742, -0.013847417017473024, 0.00826065566468616, 0.014054095203113223, 0.004627646663567003, -0.0028111419301767677, -0.0073047708023301575, 0.010043252803941652, 0.02520178208774397, 0.018471834319837482, -0.005467275453954042, 0.004543683737962168, 0.0013183786782810045, -0.011315612735840417, -0.033610986505101914, -0.033972674959786855, 0.016844245586981485, -0.004181997611583787, -0.0015638087326904219, -0.0003346405173453943, -0.030097463868476886, 0.004411280859980483, -0.005893548338721993, 0.02774650381418675, 0.009352173231887705, -0.03748619759641188, -0.02096488626703956, -0.012181076881509122, 0.026635609148447318, -0.02053861245094898, 0.010689121319159979, 0.008021684798343144, 0.005948447479165244, -0.011431869739132006, 0.015423335971088314, -0.011651464438259759, 0.005396229973159384, -0.0027998391659678633, 0.02197244137429758, -0.0027239496433693884, -0.002846664670574097, -0.001679257697189406, -0.017348024071933123, 0.002517271923390501, 0.005693329341148194, -0.01829099195514026, -0.019324380089373386, -0.0014217176546857764, 0.009998041747106032, 0.006132519670726322, -0.016288799185434393, -0.019053115611004926, -0.004995791512350472, -0.02681645337578979, -0.013208007224659783, -0.013046540095855201, 0.009507181638287199, 0.021236150745408015, -0.008738598328694818, -0.016921750255842544, 0.02350960799348234, 0.019918578825351006, 0.010262847503069402, 0.0032906990419560413, -0.009029238974278542, 0.006155125199144131, -0.0032680937463688887, 0.009533016527907551, 0.01946647198228533, -0.03454103880614413, -0.01543625341589849, 0.031854225186789396, -0.016792575807740778, -0.006051786106324031, 0.0011197742448931494, 0.003943026279579457, 0.0039236501123641925, 0.003250332259754896, -0.0004642178497940131, 0.00833816033354722, 0.012038985919919804, 0.01758053621587105, -0.015759187673507653, 8.66876409118177e-05, -0.005166946724762687, 0.00037561280477201687, -0.005987199347934461, -0.001407185645689656, -0.004288565599945118, -0.004359611080739777, 0.008628800979130942, -0.04037968660743893, -0.00860296608951059, -0.005183093530775408, -0.019479389427095505, 0.008234820775065807, 0.016715073001524968, -0.013511565315053682, 0.013640738831832823, -0.0005732081209917411, -0.0033746619675608766, -0.005108818688778205, 0.009248834604728917, -0.016882997921412016, -0.013976590534252164, 0.03567776649885866, -0.013007787761424671, 0.007834382779918209, -0.029761612166057546, 0.022153283738994802, -0.02009942305269348, 0.013040081373450113, 0.003662073484772711, -0.00395271436318709, -0.017076757730919413, -0.0074985310774988686, 0.02061611711981004, -0.014751632566249633, -0.0004844012408945859, 0.011496456031860265, -0.011735427829525906, -0.002304135248175869, -0.0073822750055299044, -0.004114181491991672, -0.005134653112737245, -0.017490114102199812, 0.024581748462145996, 0.001303039328984248, -0.018820603467066998, -0.004391904692765218, 0.030898341721417335, 0.0066653610095169175, 0.017193014734211004, 0.1845633338606527, -0.0008686929247946081, -0.005635201305163712, 0.044642417317763694, 0.01929854519975303, -0.006210023873926069, 0.019660231791792725, -0.020228595638149993, 0.005877401998370585, 0.011154145607035836, 0.006549104937547954, 0.011393117404701477, 0.006768700102337018, 0.0009655731787942917, -0.019350214978993738, -0.026997295740487013, -0.04725172626825736, 0.019285627754942854, -0.030898341721417335, -0.018562256433508718, -0.003082406641375882, -0.024736757799868114, -0.030252473206199004, -0.02380670736147115, 0.030898341721417335, 0.005990428709137005, -0.00736289883831464, -0.006374720363933196, 0.03474771792310695, 0.009765529603168104, -0.014234937567810446, 0.002958076933569901, 0.01843308198540695, 0.005134653112737245, -0.011780637955038899, -0.002543106580180198, 0.010185343765530968, -0.03309429243798535, 0.031673384684737425, 0.011625629548639405, 0.0023800247707743438, 0.006962460377505729, -0.0048407831059509784, -0.00011746730069865833, 0.0006345656345940954, 0.02204994418051339, -0.010908716018287731, -0.0063940965311484604, 0.013589069983914742, 0.03376599584282403, -0.026286841863863052, 0.0015363592788841247, 0.0513077815869098, 0.025938072716633533, -0.026118915081330758, 0.006206794512723525, 0.003930108834769281, -0.005580302164720462, 0.0017971286825902984, -0.008841937887176231, 0.009423219178343677, 0.017038007259134135, 0.006306904244341081, 0.014790384900680163, -0.028986571065382702, -0.0039075033063514725, -0.04459075126381349, 0.0065943155287222594, 0.003390808773573597, -0.011645005715854669, 0.029916621503779665, -0.02681645337578979, -0.022256623297476213, -0.024891765274944983, -0.027436487001387764, -0.023225424207658458, 0.02564097334864472, 0.006000116792744637, 8.204546328699253e-05, 0.016288799185434393, -0.0031857455013653254, -0.025072607639642204, -0.005667494451527841, -0.020603199674999864, -0.014441615753450645, -0.016818410697361133, -0.000324952491945426, 0.008021684798343144, -0.0037298896043648257, 0.016392138743915804, 0.005011938318363193, -0.01641797363353616, 0.017038007259134135, -0.01040493846465872, 0.0163404689646751, 0.041490583135823614, 0.01535874874703743, 0.014777467455869985, -0.023470855659051812, 0.03916545424586333, -0.014299524791861327, -0.013059457540665379, 0.03301678776912429, 0.0012352232093921335, -0.006578168722709539, -0.018316824982115364, -0.017980975142341274, 0.024026302060598904, 0.0012271498063857733, -0.008318784166331954, 0.003419872791565838, -0.03841624989745409, 0.012336085287908615, -0.00820898681676808, 0.018820603467066998, 0.014687046273521374, 0.0013183786782810045, -0.017063842148754487, 0.010734331444672972, 0.012045444642324894, -0.0017825965571788497, -0.016585898553423205, 0.006542646215142865, 0.02617058486057146, -0.008531921074377244, 0.0031001678951592185, -0.01539750108146796, 0.018652676684534703, 0.018019725614126552, -0.049706025881028904, 0.02598974249587424, -0.010372645783955903, -0.002299291206372053, -0.019505224316715856, -0.005270285351921474, 0.011050806979877047, 0.0009300503801892983, 0.005838649663940055, -0.006671819731922006, 0.0024171621917729452, -0.014738715121439455, -0.02077112459488691, 0.0024526849321702747, -0.005076525076752763, 0.007653539483898362, -0.0001814486266223317, 0.020009000939022244, -0.020189843303719465, -0.00946842930385667, -0.010605157927893832, -0.017218849623831356, 0.0009268210771944183, -0.01402826031349287, -0.028108190406226446, 0.01807139539336726, -0.019066033055815102, -0.03619446056597522, -0.04187810275483841, 0.00961052119676861, 0.04048302616592034, -0.030304141122794462, -0.003871980565954143, 0.021804514591765283, -0.000651923334642442, -0.011599795590341676, 0.03490272353553857, -0.16503227651696173, 0.013382391798274542, 0.00850608618475689, -0.0035651931143576994, 0.002438153039589482, 0.005961364458314107, 0.009720318546332487, 0.006135749031928866, -0.0010075546415967093, -0.0014733870846804983, 0.021817432036575463, 0.004934434115163446, -0.05637138642488451, -0.0089000654574994, -0.0002353382715476347, 0.01844599943021713, -0.008234820775065807, 0.023083334177391765, 0.01402826031349287, 0.004333776656780736, 0.011593336867936588, -0.01993149627016118, 0.019957331159781537, -0.006490976901563472, -0.004333776656780736, 0.014544954380609432, -0.008925900347119754, 0.018110147727797787, -0.02201119184608286, -0.01145770462875236, -0.0036104039383626613, -0.0064586832895380305, 0.0306399928252138, 0.021649507116688414, -0.008887148012689224, 0.007756878576718462, -0.019853991601300122, -0.021920771595056874, 0.02248913544141414, -0.00042909877954700186, 0.018316824982115364, 0.020861546708558144, 0.018265157065519906, 0.0047535908191435996, 0.006878497917562206, 0.01275589945027148, 0.005625513221556079, 0.0026302988669875773, 0.0024510702515690022, -0.015449170860708667, 0.019660231791792725, -0.009287586939159447, 0.007556659579144663, 0.009300504383969625, 0.03348181578229065, 0.013408226687894894, 0.0026464454401696415, 0.009235917159918741, -0.005880631359573129, -0.016663403222284264, 0.013356556908654189, -0.0112833200551376, 0.016056287041496465, -0.022295375631906745, 0.007724585430354334, -0.019776488795084316, -0.02096488626703956, -0.0015993314730877511, -0.00937154939910297, -0.0014992218578855233, -0.002494666394972692, -0.033972674959786855, 0.011141229093548284, -0.010379104506360991, 0.008021684798343144, 0.02411672417427014, -0.022695812695731718, 0.013459895535812977, 0.003342368588366091, -0.0036911375027649522, -0.0029128661095649397, 0.022928324839669646, 0.0018504127931862923, -0.0018907795753874378, -0.015823774897558537, 0.030019959199615826, 0.01081829483593912, -0.004362840441942321, -0.01937604986861409, -0.00572885231437618, 0.021119893742116428, -0.02787567639964327, -0.014984145641510186, -0.0028111419301767677, -0.020977803711849735, 0.018471834319837482, -0.010592240483083656, 0.006022722321162447, -0.002930627596178932, 0.00011756821823623783, 0.0037557242611545227, 0.008570672477485148, 0.012762358172676567, 0.008215445539173168, 0.014015342868682694, -0.0024559142933728187, 0.026842288265410144, -0.006348885474312842, 0.019608562012552018, 0.01466121138390102, 0.010566405593463302, -0.001814890052788963, 0.021907854150246698, 0.007563118301549751, 0.007285394635114893, 0.018898108135928057, -0.012465258804687756, -0.018988528386954043, 0.03882960440608924, -0.0007665649588407903, 0.0669636315645813, -0.0016760283359868618, -0.04335068401261754, 0.007233725321535499, -0.021003638601470087, -0.006436077761120222, -0.09398676219468866, -0.0331201273276057, 0.008273573109496337, 0.024129641619080315, -0.002822444694385672, 0.023819624806281327, -0.00813148214790702, 0.0036975962251700406, -0.028289032770923667, 0.03624613034521593, 0.018058477948557083, -0.015746270228697477, 0.0007096478168919342, -0.014428698308640468, 0.0225666401102752, -0.006109914607969826, -0.005121735667927069, -0.007711667985544157, -0.001945678366584704, 0.009959289412675503, -0.015759187673507653, -0.009436136623153853, 0.015009979599807913, -0.01663756833266391, 0.0006426389793927918, -0.014415780863830292, -0.019634396902172373, 0.02283790458864366, -0.0018229633393799952, 0.01758053621587105, -0.002559253153362262, -0.0036911375027649522, 0.01701217236951378, -0.0331201273276057, -0.0006660517316959086, -0.007666456928708539, -0.01772262624613774, -0.0009809125280910481, 0.016237129406193686, -0.017063842148754487, -0.014790384900680163, 0.003949485001984546, 0.008544837587864795, -0.016844245586981485, 0.015009979599807913, -0.023315846321329693, -0.02414255906389049, -0.003497376994765585, 0.0017938993213877544, -0.005880631359573129, -0.015578344377487808, -0.0031970482655742302, -0.02140407566529506, 0.005422064397118424, 0.021817432036575463, -0.006884956174305982, 0.0405605308347814, 0.014092846606221128, -0.0005691714776962251, 0.0089000654574994, 0.019001445831764222, 0.008790268107935525, -0.02314792140144265, 0.037847882325806326, -0.012523387306333551, 0.006872039195157118, -0.0023073646093784133, -0.030898341721417335, 0.03565193160923831, -0.00807335457758385, -0.01167729932788011, 0.021468662889345944, -0.028211528102062608, -0.0030242783725607436, -0.028573214694102303, 0.0028111419301767677, -0.01758053621587105, -0.024736757799868114, 0.02159783733744771, -0.0227603999197826, -0.01472579767662928, -0.025175947198123615, 0.02681645337578979, 0.006613691695937524, 0.012103572212648062, -0.007627705059939322, 0.014273689902240976, 0.014015342868682694, -0.0013345254842937254, 0.011683758050285199, -0.011470621142239911, 0.03771871160299506, 0.021507415223776475, -0.02291540925750472, -0.01911770283505581, -0.001089095429884308, 0.0004127502257706182, -0.014441615753450645, 0.0347735490874368, -9.607291369904755e-05, -0.011037889535066871, -0.0178001309149988, -0.0302008034269583, 0.041335573798101495, 0.006574939827168307, -0.016624650887853733, 0.019660231791792725, -0.006206794512723525, 0.008409205348680567, -0.015113319158289326, -0.004456491451154789, -0.008674012035966561, -0.009261752049539095, 0.017554701326250696, 0.0016049828551922033, -0.01835557731654589, -0.027694834034946048, -0.01402826031349287, 0.012872155522240443, 0.012626725933492337, 0.00466639853233622, 0.03443769924766271, -0.0012247277854838651, -0.0007540513005962598, 0.028082355516606095, 0.009759070880763016, -0.014222021054322893, -6.130703300569051e-05, -0.018639759239724527, 0.014183268719892363, -0.024698005465437583, -0.018265157065519906, 0.012852779355025178, -0.033223466886087114, -0.01990566138054083, 0.04146474824620326, -0.005183093530775408, 0.015449170860708667, 0.012684854435138133, -0.00298552627096087, 0.012458800082282668, 0.020551529895759157, -0.048698472636416135, -0.023742120137420268, 0.011251026443112159, -0.023238341652468634, -0.011425411016726918, -0.007401651172745169, -0.024517161238095112, 0.03087250683179698, 0.015940030038204878, 0.03043331557089623, -0.009061532586303982, 0.012297332953478086, 0.012071279531945246, -0.005783750989158117, -0.004408051498777939, -0.031001679417253496, 0.017980975142341274, 0.0038816686495617755, 0.012607349766277073, -0.029684109359841736, 0.027281477663665645, -0.006487747540360927, 0.013653656276643, 0.0022476218927926592, 0.010398479742253632, -0.002244392531590115, 0.009513640360692287, 0.018381412206166247, -0.015526674598247102, -0.03490272353553857, -0.012936742746291325, -0.012910907856670973, -0.009455512790369118, 0.03195756474527081, 0.031337531119672836, 0.021236150745408015, 0.007982932463912615, 0.0008012804171451476, -0.034127684297508974, 0.033455980892670295, 0.014015342868682694, -0.01299487124793712, 0.003001673076973591, 0.009655731322281605, 0.02527928675660503, -0.002930627596178932, -0.011076641869497401, 0.01322092466946996, -0.007950638851887173, 0.0331201273276057, -0.017193014734211004, -0.006988295267126082, 0.002543106580180198, -0.01402826031349287, 0.00957176886233808, 0.026093080191710402, 0.022502052886224318, -0.012930284023886237, 0.00649743562396856, 0.021507415223776475, -0.015513757153436924, -0.011845225179089782, -0.0002458336372482391, -0.01578502256312801, -0.0033197632927789386, 0.039707983202600244, -0.03753786737565259, -0.029270752988561338, -0.003079177280173338, -0.008157317037527373, 0.03738285803793047, 0.0031356906355565475, -0.005018397040768281, 0.019091867945435458, -0.027281477663665645, -0.008654635868751296, 0.023690450358179564, -0.0012901220005893997, -0.03193172985565046, 0.030743332383695216, 0.014041177758303047, 0.0011779023972929596, 0.006671819731922006, -0.014273689902240976, 0.006846204305536765, 0.007485614098350004, 0.005554467740761421, -0.003200277626776774, -0.01949230687190568, -0.007731043687098109, 0.009694483656712135, -0.002701344347782236, -0.022450383106983614, -0.010204719932746234, -0.010237012613449049, 0.001220691200396013, -0.005108818688778205, 0.029658274470221384, -0.009087367475924335, 0.028237362991682963, 0.007472696653539828, -0.0039171913899591045, 0.02595099016144371, -0.005196010509924271, 0.02551179890054296, 0.009894703119947246, 0.017774296025378448, -0.021507415223776475, -0.0153199964126069, -0.006332739133961434, -0.01809723028298761, 0.01896269349733369, -0.019208124948727045, -0.023483773103861988, 0.004337006017983281, -0.03265510303972985, 0.016611733443043557, 0.014441615753450645, -0.022669977806111363, -0.012394212858231784, 0.01829099195514026, 0.009901161842352334, -0.0009849492295942283, -0.0028256740555882166, -0.011612712103829229, 0.016895915366222192, -0.010534112912760485, -0.0065135819643199686, -0.021843266926195815, 0.014338277126291857, 0.004814948216330626, -0.017193014734211004, -0.0021539708835801915, 0.02854737980448195, -0.023690450358179564, -0.002087769444589349, -0.0016889456643817103, 0.01466121138390102, 0.033404311113429584, -0.019711901571033433, -0.0021814202209711605, -0.008880689290284136, -0.03591028050544184, 0.04404821858178607, 0.021494497778966296, -0.006310133605543625, 0.013718243500693882, -0.014544954380609432]', '[0.006067243567406174, -0.02555262365922449, 0.0008215014461026088, -0.018800098953198813, 0.003068721702667197, 0.020992998596781227, -0.005809845558653267, -0.008363770545322701, -0.013906189613470643, -0.0034732046537270717, 0.016620571239501353, 0.01894718273135253, 0.00453622558212849, -0.018653013312399894, -0.0005135428647403989, 0.007006580197968647, 0.053271403102261486, -0.00903233723724731, 0.01853267153137089, -0.030005278870523705, -0.005475562212690963, 0.008337028548197989, -0.0012351771915047535, -0.005562476031652778, -0.003967944173389351, -0.015163095143300527, 0.017663535204397934, -0.0231992683076034, 0.008096344054817376, -0.006755867688496918, 0.012308314772735978, 0.001142413502464245, -0.03695837134895253, -0.01722228014464638, 0.0014031546101037158, -0.03005876286477313, 0.011592948188859222, 1.0168875223700575e-06, 0.005893416395143843, -0.02721066845915106, 0.007828917564312053, -0.012187972991706974, 0.007320806580426118, -0.013986417467489982, -0.014614870232404922, -0.00015847121965001368, -0.013491678413489002, -0.01033604265902934, -0.025084626602348226, 0.008310285619750677, 0.018131532261274202, 0.008898624457655949, 0.004753510129533027, -0.01628628789353905, 0.008631197035828025, 0.022784757107621758, -0.01360533422957553, 0.004232027681423436, 0.001905415374664983, -0.022905098888650763, 0.002157799258957006, 0.0056193039396960414, -0.015136352214853214, 0.01827861603942792, -0.008443998399342038, 0.009674161932047207, 0.006154157386367989, 0.013050424285060048, 0.004168514274098994, 0.01156620619173451, 0.02840071806484656, 0.020645343320933967, -0.006231042723577387, -0.01765016327451298, 0.013591963231013174, 0.0021260425552947853, -0.020645343320933967, -0.009460220367113909, -0.009954959421114889, 0.020230832120952325, 0.020257574118077038, -0.011813575718735, -0.016152574182625085, 0.01050987029695297, -0.011867060644307025, -0.017142052290627045, -0.014494528451375917, 0.009052394200752146, -0.004773567093037862, -0.009206164875170942, 0.02710369860800701, 0.013391393595964831, -0.006418241360063374, 0.009085823094141935, -0.00738097747094062, -0.0001059260567886405, 0.006354727487077633, 0.026555473697111406, 0.00281132346108364, -0.02791934907808534, -0.006816038579011421, -0.0010295928499189021, -0.014253843957995305, -0.005569161530933956, -0.034792215565140026, -0.02330623815874745, 0.009694218895552041, 0.0011256994050153249, 0.02920299846768513, -0.030486645994639728, -0.014280586886442619, 0.028801857334943247, -0.004964108711995088, -0.028694887483799197, 0.004616453901809127, -0.017877476769331234, -0.0034230620121343362, -0.002841408906340892, 0.006190928796567719, -0.02467011540236658, 0.008263485727798529, 0.009319821622580068, 0.03866990386841892, -0.013250993920108391, 0.006859495488492328, 0.011379006623925924, -0.02520496838337723, -0.008363770545322701, 0.0008085479711031825, 0.0206052293939243, 0.03634329237656774, -0.00025384646822702156, 0.02147436572089725, -0.01909426837215145, -0.025820049218407217, 0.03743974219835895, -0.022891726958765807, -0.0015101252761550404, -0.03118195654633425, -0.020792427099087685, -0.0005402855603570613, 0.05155987058279509, -0.0027962807384550145, -0.0021394137866877915, -0.015978746544701456, 0.0025957106377453715, 0.003533375544241575, 0.0040548575266898665, 0.0009343220986479517, -0.00432562746532773, 0.024910798964424594, -0.014962525508252184, 0.007434462862173945, -0.002094285618801914, 0.01592526068780683, 0.005305077557238571, -0.00910588005764677, 0.009647419003599896, -0.016968224652703413, -0.01420035903242328, -0.003209120680031687, -0.026983356826978003, 0.0099817023495622, -0.005866673932357831, -0.0020608571910734237, 0.015952004547576743, 0.020658715250818922, 0.020551743537029675, -0.021354023939868245, -0.0071603504067261434, -0.016727541090645403, 0.016259544033769134, -0.022236532196726156, -0.0029082655289672226, 0.0008210836315013664, 0.007701889818340569, 0.011439178445763025, 0.006919666379006832, -0.028320490210827224, -0.007019951662192303, -0.00425542762739951, 0.02917625460791522, 0.023640521504709754, 0.017008340442358282, -0.014227101960870592, 0.013759104903994327, 0.030326190286601055, -0.01069038296849648, 0.0013705619279930352, -0.0024419401961572255, 0.022437101831774497, 0.010035188206456824, 0.015965374614816497, -0.0169414826555787, -0.6816172186551559, -0.02511136859947294, -0.005732960221443869, -0.014039903324384605, -0.0005139607375493038, 0.01637988767744334, 0.03420387672723475, -0.005057708030238081, -0.034685247576641175, 0.016540343385482013, -0.016058974398720793, 0.023319610088632408, -0.016513599525712103, -0.006378127433053706, 0.0010697068933438957, -0.002271455773535484, 0.004803652538295114, -0.02485731310752997, 0.00013935439720563723, 0.012000774355220987, -0.0065084976958351285, -0.00043206130498285334, -0.019040782515256822, -0.009921531459047698, 0.008831767602198967, -0.0012293272050107353, 0.002826366183712266, 0.007614975999378754, -0.012896653610641252, 0.010717125896943791, -0.017342623788320587, 0.026033990783340514, 0.00041827211346944854, -0.018746613096304186, 0.04492768765779842, -0.0013062124258048085, -0.006140785922144333, 0.025846793078177128, 0.010870896571362587, 0.05468876434013004, -0.012662655082203118, -0.002833051915824094, 0.011760089861840375, -0.008443998399342038, -0.014240472959432949, 0.02425560420238494, 0.03575495353866247, -0.009440163403609075, -0.024843943040290212, -0.016500229458472345, 0.03203772094342225, -0.0022346845961664042, 0.010202329879437977, 0.008183257873779192, 0.012428656553764985, 0.0014858897009764824, 0.024656743472481625, -0.02249058768866912, 0.01802456054748495, 0.009440163403609075, 0.0031071641384412462, -0.0007947588086936089, -0.012542313301174113, -0.017596679280263552, -0.005502304675476975, 0.01866638524228485, -0.005037651066733247, -0.02401492064032693, 0.013518420410613715, -0.0022731270319404535, -0.0005260785251385828, 0.019709349207181433, -0.009186107911666106, -0.012027517283668299, 0.01262254115519345, 0.018104788401504292, 0.006414898377592135, -0.016406629674568053, -0.014828811797338222, 0.009373306548152093, -0.009018966238684955, -0.016620571239501353, -0.006541926123563619, 0.0037807453040727143, 0.05717583154001989, 0.0011499348637785578, -0.03160983781355565, -0.011760089861840375, 0.009025652203627433, 0.009152679949598915, 0.01644674360157772, 0.025779935291397545, 0.005873359431639009, -0.007675146889893257, 0.013097224177012194, 0.006144128904615572, -0.015524121417710144, -0.013264365849993347, 0.022156304342706816, -0.017489707566474305, -0.008103030019759854, -0.01254899926611659, 0.025084626602348226, 0.021033112523790896, 0.013157395067526697, -0.002405169018788146, -0.013632077158022842, 0.007581547571650264, 0.023961434783432305, -0.025993876856330846, 0.0015878461261515982, -0.01873324116641923, -0.006127414923581977, -0.012963510466098232, 0.011124952063305556, -0.04016749296030681, 0.00207255716406146, 0.01349836344710888, 0.011191807987439937, -0.022971954812785144, 0.015216580068872552, 0.00347989015300825, 0.01771702106129256, -0.010021816276571869, -0.003947887209884517, 0.006859495488492328, 0.008671311894160294, -0.025739821364387877, -0.004215313700389841, -0.008089658089874899, 0.020551743537029675, -0.009293078694132756, 0.011439178445763025, -0.012769625864669768, 0.020163974334172743, -0.018158274258398915, 0.0016730884538777942, -0.025860165008062083, 0.023012068739794816, -0.0066956967979824155, -0.013973046468927625, 0.011699918971325872, 0.00734754904321213, 0.011626377082249013, -0.0022697842822998644, -0.019040782515256822, -0.03730602662479979, -0.019722721137066388, -0.018933812664112773, -0.014949153578367227, -0.005361905930943134, -0.007755374743912594, -0.01628628789353905, 0.020391287828991, 0.018960554661237486, -0.034498048008832584, -0.009186107911666106, -0.011238607879392083, -0.02295858474554539, -0.017035082439482995, 0.01317745203103153, 0.020337801972096375, -0.017142052290627045, -0.012181287026764494, 0.016259544033769134, -0.02626130427815877, 0.003055350238443541, 0.025579365656349204, 0.010496499298390614, -0.028454203921741184, -0.0063480419877964545, -0.037573454046627715, -0.010523241295515327, 0.027304268243055353, 0.013411450559469665, 0.01830535989919783, -0.02717055453214139, -0.007695203853398091, 0.021634821428935927, 0.005124564420033762, -0.010911010498372255, -0.023185896377718444, -0.02840071806484656, -0.001870315688531523, 0.01613920225274013, 0.006535240624282441, 0.018706499169294517, 0.034658503716871264, -0.013558535268945984, 0.018826840950323526, -0.014735212013433929, 0.022784757107621758, 0.004646539347066378, 0.0048604809119996775, -0.010188957949553022, 0.008417256402217325, -0.012368485663250483, 0.017917590696340902, 0.014414300597356579, 0.01648685752858739, 0.04230690860963981, -0.02105985452091561, 0.025820049218407217, -0.0028915515479336273, -0.0032642773296699813, -0.03439107629504334, -0.011666491009258682, -0.012348428699745647, 0.016393257744683098, -0.006632182924996673, -0.01518983807174784, -0.044606776241721074, -0.028240262356807887, -0.024108518561586023, 0.020525001539904962, 0.012141173099754826, 0.023533551653565705, 0.01989654877499002, -0.027290896313170398, -0.009954959421114889, 0.008557655146751166, -0.022570815542688458, -0.014708470016309216, -0.004813681020047531, -0.0009827931907974055, 0.004188571237603828, 0.019950032769239446, 0.024977656751204173, 0.008337028548197989, -0.010944438460439446, -0.01702171050959804, 0.009553819219695602, 0.0015819962560729049, 0.01978957706120077, 0.002433583205640428, -0.0060104156593629105, 0.026702557475265124, -0.025191596453492275, 0.03883035957645759, 0.01153277729834472, 0.020591857464039343, 0.009273021730627922, 0.021982476704783187, 0.001500932540020433, 0.008136457981827045, -0.004720081701804537, 0.036477004224836504, 0.0032542488479175642, -0.013852704687898619, 0.009366620583209615, -0.00650181219655395, -0.0003495350923068768, -0.002784580765466978, 0.01747633563658935, 0.013973046468927625, -0.023453323799546368, -0.009988388314504678, 0.00867799692778017, 0.03420387672723475, 0.03751997005237829, 0.010743868825391103, -0.00049223228280885, 0.020779057031847927, 0.0076617754256696, 0.009266335765685444, -0.011746718863278018, -0.003158978271269601, -0.010235757841505167, -0.026769415262044707, -0.011793518755230164, -0.011078152171353408, -0.028561173772885237, 0.01486892572434789, -0.02376086514838396, 0.004439283747075557, -0.00140566173054182, 0.0005411212477672085, 0.00808297305625502, 0.014414300597356579, -0.01761004934750331, -0.008597769073760834, -0.010362785587476652, 0.01950877957213309, 0.01644674360157772, -0.0022162988910665394, -0.019348323864094415, -0.02541890994831053, -0.006384812932334884, -0.01420035903242328, 0.0036269748624845684, 0.008076287091312542, 0.027544951805113366, 0.01613920225274013, 0.0033261199442507538, -0.00039069373044263884, -0.01853267153137089, 0.030486645994639728, 0.00362028936320339, 0.00762834746360241, -0.008992223310237642, 7.777312047746007e-05, -0.016607199309616395, -0.0059001018944250215, -0.027678665516027327, 0.026167704494254478, 0.01198740242533603, -0.012682712045707952, -0.03139589811126755, -0.018800098953198813, 0.007454519825678779, -0.01711531029350233, 0.0028681516019575543, -0.014253843957995305, 0.004305570036161595, -0.0020825856458138777, -0.01803793247736991, -0.032920231062925354, 0.0007237235743935541, 0.009366620583209615, -0.006267813668115817, -0.027892607080960627, -0.01240859959026015, -0.007093494016930462, 0.008283542691303363, 0.07926528368388273, 0.04709384716690132, 0.0028631373610813454, 0.006548612088506097, 0.01216791602820214, 0.01346493548504169, -0.004843766465304782, -0.03390970917092731, -0.006475069733767938, -0.008230057765731338, 0.011853689645744668, -0.019121010369276163, -0.0036169463807321513, -0.013451564486479333, 0.015216580068872552, 0.022624301399583085, 0.0029684366523123756, -0.009045709167132267, 0.01332453674050785, -0.004342341446361325, 0.0014649969918538383, -0.004984165675499922, 0.0018569443407231918, 0.01754319342336893, 0.010068616168524017, 0.017877476769331234, 0.025071254672463267, 0.014989267505376895, 0.005104507456528928, -0.0038944015858205423, -0.005181392793738326, 0.006317956542539202, 0.008891938492713471, -0.007340863543930952, -0.01644674360157772, 0.02337309594552703, -0.0016689098422040703, -0.004459340710580391, 0.04653224846347556, -0.008316970653370554, 0.029898307156734454, 0.027812379226941287, 0.0005457176740421748, -0.00022313417010066592, 0.019709349207181433, -0.001659717106069463, -0.022758013247851847, 0.02939019617284852, 0.002478711373526305, -0.013304479777003015, 0.037867625328225546, 0.006719096278297189, -0.009767760784628902, -0.014240472959432949, -0.0025221682830072127, 0.022971954812785144, -0.004619796418619066, -0.0011190136729034967, -0.004713396202523359, -0.004526197100376073, -0.014521271379823229, -0.02654210176722645, 0.015724691052758488, -0.004248742128118332, -0.003670431771965476, -0.03054013185153435, -0.03789436546270506, -0.01637988767744334, -0.03313417076521345, -0.009747703821124068, -0.015229951998757508, -0.002172842214416282, -0.0028982370472148055, 0.008918681421160783, 0.012602484191688615, 0.014641613160852236, 0.004810338037576292, -0.006070586549877413, 0.015096238287843546, -0.0026023963698572, -0.0036336605945963964, -0.011559520226792032, -0.005488933676914619, -0.010997924317334071, -0.024389317913298902, 0.006732467742520845, -0.015350293779786514, -0.009513705292685934, -0.006027129640396506, 0.013919560612033, 0.005839931003910519, 0.00829691368986572, 0.015096238287843546, -0.014227101960870592, 0.001082242495534417, -0.018345473826207503, 0.01543052163380585, 0.018639643245160136, -0.011292093736286708, 0.0009326507238276571, 0.02172842121284022, -0.008764910746741987, -0.0031907349749318225, -0.005773074148453538, 0.015738062982643443, 0.020270946047961994, 0.014467785522928605, 0.02101974059390594, -0.0026993384377407823, -0.017315879928550673, 0.029229740464809844, -0.02893557104585721, 0.015671205195863864, 0.005662760849176949, 0.0007024129924620052, -0.005512333157229392, 0.006491783714801533, 0.02874837334069382, 0.007320806580426118, -0.013016995391670257, -0.01233505770118329, -0.02660895955400603, 0.005559133049181539, -0.0017249023538754994, 0.002889880056698008, 0.00875153974817963, 0.011639748080811368, -0.01518983807174784, -0.011218550915887249, 0.008637883000770502, -0.009747703821124068, 0.01950877957213309, -0.0004337327089069791, -0.02807980664876921, -0.029684365591801157, -0.019268096010075078, -0.017797248915311897, -0.005395334358671625, 0.0025606109516119117, -0.024843943040290212, 0.0006480919720261956, -0.003817516481441794, 0.000413884652702766, -0.038054821170743734, -0.00038693304978548243, -0.06354058704318864, 0.013083852247127239, 0.007327492079707296, 0.008992223310237642, 0.010489813333448136, 0.006147471887086811, 0.00030085506375297067, -0.01687462673144432, 0.01050987029695297, -0.01022907187656269, -0.018706499169294517, 0.008056230127807708, -0.012027517283668299, 0.014080017251394275, 0.018786727023313857, 0.029309968318829184, 0.011338892696916255, 0.0072606356899116145, -0.011225236880829726, 0.013852704687898619, 0.002217970382302159, -0.01922798208306541, 0.0017349308356279165, -0.028775115337818534, 0.03955241398792203, 0.009259650732065566, 0.021193568231829572, 0.010743868825391103, -0.021033112523790896, -0.01212780210119247, -0.006301242095844308, 0.002064199940714013, -0.011579577190296866, 0.005816531057934445, -0.026234562281034057, -0.024295718129394607, 0.0034732046537270717, -0.005258278130947724, 0.0014081688509799243, -0.00026742672333597393, 0.01223477288365912, 0.0073141210811449395, 0.004345684428832564, 0.014561385306832897, -0.010422956477991156, 0.0217952771369746, 0.0021895561954498773, 0.019602379356037383, 0.010951124425381925, -0.004679967774794869, 0.0024586544100214706, -0.02354692358345066, 0.002453640169145262, 0.013825961759451307, 0.010730496895506147, 0.008972166346732808, 0.01735599385556034, -0.02361377950758504, 0.013130652139079385, 0.003984658154422946, 0.004128399881428026, -0.0336690256088693, -0.02159470750192626, 0.0017382737016838304, -0.024763715186270876, -0.01141912148225819, -0.003921144281437205, -0.026662443548255456, -0.03564798182487322, 0.008938738384665617, 0.008798339640131776, -0.009453534402171431, 0.009787817748133736, -0.001142413502464245, -0.024977656751204173, -0.01866638524228485, -0.014293957885004973, 0.013418135593089544, 0.005144621383538597, 0.021434251793887582, 0.011974031426773673, -0.007588233070931442, -0.03770716962018687, 0.01702171050959804, -0.00019440669505703032, 0.021394137866877914, 0.023814349142633386, 0.007033322660754659, 0.005763045666701121, -0.0092797076955704, 0.009059080165694624, 0.02537879602130086, -0.02190224885076385, -0.029096026753895884, 0.007976002273788372, 0.009286392729190279, 0.00456965400985698, 0.0018619585815994003, -0.011980717391716153, -0.02818677649991326, 0.024202118345490315, -0.001810144681601695, -0.01272282597271762, -0.02214293241282186, -0.006284528114810712, -0.010155529987485831, 0.01420035903242328, -0.00868468289272265, 0.020511629610020007, 0.010061930203581537, -0.023466695729431324, -0.02147436572089725, 0.002573982183004918, -0.008891938492713471, -0.01430732981488993, 0.0006531062129024042, 0.023787607145508673, -0.010349414588914295, -0.007280692653416449, 0.0007742839141722073, -0.013591963231013174, -0.019254724080190123, -0.007394348935164276, 0.008203314837284027, 0.02791934907808534, -0.026863015045949, -0.002639167547226279, -0.001995671826852062, -0.005328477503214644, 0.0062377282228585655, -0.01226820084572631, -0.008617826037265668, -0.008597769073760834, -0.016152574182625085, -0.002288169987399729, 0.025311940097166478, 0.013665505120090033, -0.0137056199784223, -0.009413420475161763, -0.010670326004991645, -0.022089446555927237, 0.008985538276617763, -0.0047501671470617885, -0.007481262754126092, 0.011853689645744668, -0.013250993920108391, -0.0013998117440478018, 0.008584398075198478, 0.009954959421114889, -0.010610155114477143, -0.023948062853547347, 0.003687145985829721, 0.02337309594552703, -0.022758013247851847, 0.018586157388265512, -0.009065766130637101, 0.004425912282851901, -0.033989937024946654, 0.02920299846768513, -0.011479292372772695, -0.02928322632170447, -0.0094936483291811, -0.008383827508827536, -0.013839332758013663, 0.0032141349209078955, 0.00685280998921115, 0.028828601194713158, -0.0020692141815902214, 0.008484112326351706, 0.013378021666079874, 0.003606917898979734, -0.00012713215596604757, -0.021741793142725174, -0.015109610217728501, 0.0308610432676117, -0.005047679548485664, -0.013264365849993347, 0.004068228990913522, -0.0018853584111601486, 0.01329110784711806, 0.024509659694327907, 0.012856539683631582, 0.01884021288020848, -0.034123648873215416, -0.029149512610790507, -0.011786832790287687, -0.0016338101560706103, -0.020765685101962972, -0.01016221502110571, 0.0078757165249416, 0.0025572679691406724, -0.013518420410613715, 0.017583307350378597, 0.022584187472573417, 0.001120685164139116, 0.009266335765685444, 0.006200957278320136, 0.0023917977873951397, -0.0032575918303888035, -0.0008825081986502465, -0.023346352085757117, 0.006374784450582467, -0.03219817665146092, -0.0036503748084606414, -0.021140082374934945, -0.001005357391155669, 0.009754389786066546, 0.00977444674957138, -0.02710369860800701, -0.021567965504801546, -0.00960061911164775, -0.024268974269624696, -0.013525106375556192, -0.010944438460439446, 0.02682290111893933, 0.02973785144869578, 0.0071269219789976525, 0.010275871768514837, 0.029711109451571068, -0.0057195892228815125, 0.017623421277388265, 0.003687145985829721, -0.020538371607144716, 0.016928110725693744, -0.025151482526482607, -0.00812308698326469, 0.016259544033769134, -0.01635314381767343, -0.013973046468927625, 0.03441782015481325, 0.021527851577791877, 0.004857137929528439, 0.027304268243055353, 0.0040648860084422835, 0.0049875081923098615, -0.004666596310571212, -0.00653858314109238, 0.00516802132951467, 0.00043498626912603125, 0.023185896377718444, -0.02591364900231151, 0.014347443741899599, 0.02376086514838396, -4.188989008549324e-05, -0.023212640237488355, -0.02172842121284022, 0.003794116535465721, -0.00752137668113576, -0.004138428363180443, -0.030486645994639728, -0.01765016327451298, -0.006334670523572798, -0.01708856829637762, 0.01743622170957968, -0.00022793948912433766, -0.012782997794554723, 0.00829022865624584, 0.0018669728224756088, -0.006648896906030269, 0.009373306548152093, 0.008738167818294675, -0.011626377082249013, 0.010182272915933143, 0.023145782450708776, -0.013230936956603555, 0.003887716086539364, 0.0009410077725521173, 0.0014875611922121018, -0.020190718193942657, 0.0005966958866297328, 0.0034046765398651215, -0.018251874042303207, -0.007621661498659932, 0.027518209807988653, 0.03219817665146092, -0.012154544098317182, 0.0027544953202097264, 0.011893803572754337, -0.021460993791012295, -0.023841093002403297, -0.023493437726556037, -0.005221506720747995, -0.012869910682193939, -0.02380097907539363, -0.0002833051915824094, 0.007675146889893257, -0.026074104710350182, 0.0034999473493437343, -0.013117281140517028, 0.007955945310283537, 0.020137232337048033, 0.21993174248691152, -0.026876385113188756, -0.008577712110256, 0.0062243567586349096, 0.010590098150972307, -0.011151694060430269, 0.02162145136169617, 0.005609275457943624, -0.0015268394900192857, 0.00829022865624584, -0.01592526068780683, -0.004937365783547775, -0.0054789051951622016, 0.0038743446223157077, -0.004703367720770942, -0.02035117390198133, -0.02995179301362908, -0.009988388314504678, -0.04794960970134412, -0.010289242767077194, -0.00882508163725649, -0.010677011969934123, -0.0028631373610813454, -0.005258278130947724, 0.024616629545471957, -0.006354727487077633, -0.004409198301818306, 0.009112566022589247, 0.028347232207951937, 0.009326506656199947, -0.021166824372059658, -0.015631091268854192, -0.007795489136583562, 0.005188078293019504, -0.00994827438749501, -0.012121116136249992, 0.008464055362846872, -0.013919560612033, 0.013244308886488513, 0.002166156482304454, 0.006812695596540182, 0.023854463069643055, -0.0034147050216175386, -0.006063901050596235, -0.004626482383561544, 0.02397480485067206, -0.010382842550981486, -0.010610155114477143, 0.011017981280838906, 0.012014145353783343, -0.04236039260388923, -0.015296808854214488, 0.037840881468455635, 0.03845596416613082, -0.009259650732065566, 0.001560267801332451, 0.02636827412930282, 0.007902459453388912, -0.011646434045753848, 0.009266335765685444, -0.01799781855036024, 0.010075302133466494, -0.020832541026097353, 0.03284000320890602, -0.010723811861886269, 0.018198388185408584, -0.018345473826207503, -0.022784757107621758, 0.015631091268854192, -0.0010605140407939633, 0.0010630212776473926, -0.03318765475946288, -0.0016822811900124015, 0.0003606082269777249, -0.012769625864669768, -0.02101974059390594, 0.030219220435457002, 0.008457370329226993, 0.003143935315810326, 0.008517541219741498, 0.0008026980428168266, -0.025539251729339536, 0.012916710574146086, -0.003924487263908444, -0.01702171050959804, -0.030941271121631038, 0.001599546099139635, 0.005051022065295604, -0.008263485727798529, 0.006421584342534613, 0.00562264692216728, -0.028641401626904574, -0.005295049075486153, -0.011900489537696814, 0.005438790802491233, 0.03152960995953631, 0.007354235008154608, 0.04610436533360897, -0.014106759248518987, 0.023680635431719423, -0.002211284650190331, -0.02713044060513172, 0.00031777815581400585, 0.02092614081000165, -0.004171856790908934, -0.005381962894447969, -0.008517541219741498, 0.015403779636681138, 0.0035935466675867276, -0.002199584910032944, 0.014320700813452287, -0.03554101011108397, 0.00924627880218061, 0.009794503713076214, 0.027544951805113366, 0.010817410714467963, -0.006699039314792354, -0.013839332758013663, 0.003907772817213548, -0.010469756369943302, 0.018719871099179476, -0.01367219108503251, -0.00011292511347161132, 0.017797248915311897, -0.002456982918785851, 0.01093106746187709, -0.025365424091415904, 0.0007212164539554498, 0.006836095542516255, -0.04725430287494, 0.01676765501765507, -0.005194764257961982, 0.01627291596365409, -0.009888103496980508, -0.00790914541833139, 0.011893803572754337, -0.00748794825340727, 0.0012176273484380236, -0.009727646857619232, 0.0037105456989751447, -0.013337907739070206, -0.014882296722910247, 0.0036938317179415494, 0.02759843766200799, 0.004242056163175853, -0.015631091268854192, 0.03599563710072048, -0.012943453502593398, -0.015283436924329532, -0.0014181973327323416, -0.012047574247173133, -0.007013265697249825, -0.015096238287843546, 0.002833051915824094, 0.027437981953969313, 0.0014315686805406727, -0.033989937024946654, -0.04150462634415603, 0.015604349271729481, 0.025779935291397545, -0.01894718273135253, -0.0010981209637808526, 0.013090538212069716, -0.01543052163380585, -0.02239698790476483, -0.004489426155837643, -0.17425525655787874, 0.033989937024946654, 0.0063580704695488715, -0.018238502112418252, 0.021153454304819904, -0.002182870696168699, 0.02087265495310702, 0.009573877114523037, -0.019321580004324504, 0.0022062704093141223, -0.0027645238019621435, 0.014614870232404922, -0.026903128972958667, -0.01802456054748495, 0.007401034434445454, 0.015978746544701456, -0.016981596582588368, 0.017516449563599018, 0.011479292372772695, 0.0013454907236119923, 0.0047835955747902795, -0.01458812730395761, 0.036263064522548405, -0.00019242188652622588, -0.0035701467216106546, -0.0012184629776405085, -0.006979837269521335, 0.0004358219856400098, 0.011933917499764005, 0.0005599247092606532, -0.01702171050959804, -0.011639748080811368, 0.04300221543604393, 0.02067208531805868, -0.01814490232851396, -0.008062916092750186, -0.020939512739886604, -0.008925366454780662, -0.007882402489884078, 0.02440268798053866, 0.017489707566474305, 0.023747493218499005, 0.00030774964495775746, 0.00014113026933198922, -0.0130036243931079, 0.005231535202500412, 0.01933495193420946, 0.011780146825345209, -0.00016536581540671608, -0.003663746039853648, -0.004191913754413768, -0.013264365849993347, 0.017623421277388265, 0.009707589894114398, -0.0006577025809697079, -0.019990146696249114, 0.0041317428638992645, 0.01022907187656269, 0.005997044195139254, -0.005375276929505491, -0.023894576996652723, -0.019709349207181433, 0.0035868609354748997, -0.007748689244631415, -0.0017416165677397446, -0.01726239407165605, -0.017235652074531337, -0.02207607648868748, -0.03562123796510331, 0.011285407771344229, 0.003387962326000876, -0.002594039146509752, -0.0002814248512538312, -0.004994194157252339, 0.023078926526574395, 0.011292093736286708, -0.011412435517315713, 0.02963088159755173, 0.007835602597931932, -0.012562370264678947, -0.007762060708855072, 0.017676907134282892, 0.00013423567357528682, -0.008143143946769524, 0.01145923540926786, 0.02421549027537527, 0.013144023137641741, 0.009460220367113909, -0.007775432173078728, -0.023533551653565705, 0.024202118345490315, -0.01276293989972729, -0.00960061911164775, -0.005843273986381758, 0.009119251056209126, 0.02421549027537527, -0.0009017294747449335, 0.013839332758013663, 0.006448326805320626, -0.0025890249056335435, 0.01568457712574882, 0.0008833438860603938, -0.023560293650690418, 0.005278335094452558, 0.001610410326509862, -0.016633941306741108, 0.02721066845915106, 0.033936449305406825, 0.024068404634576355, -0.002478711373526305, -0.007354235008154608, 0.010422956477991156, 0.020725571174953304, 0.01754319342336893, 0.009319821622580068, 0.014949153578367227, -0.019589007426152428, -0.0132175659580412, 0.011693233006383395, -0.017930962626225858, 0.045970653485340206, 0.024135262421355934, -0.021046482591030653, -0.004211970717918602, -0.01357190626750834, -0.013545163339061026, -0.09723635278653746, -0.04471374795551032, 0.011198493952382415, 0.036477004224836504, -0.0036269748624845684, 0.026809529189054375, 0.004830395001081126, -0.0004168096168459439, -0.03883035957645759, 0.031904009095153486, -0.0006154992899155149, -0.009553819219695602, -0.014815440798775866, -0.004880537875504512, 0.008316970653370554, -0.00576638864917236, -0.028133290643018637, -0.0077888031716410846, -0.02354692358345066, 0.03297371505717478, -0.004900594839009346, 0.010937753426819569, -0.025499137802329867, -0.005699532259376678, 0.0004266291912977399, 0.021674935355945595, -0.009306449692695113, 0.027758893370046663, 0.021300538082973622, 0.007675146889893257, 0.012996938428165423, -0.013839332758013663, 0.026849643116064043, -0.03642352023058708, 0.00015074092193124837, -0.014106759248518987, -0.05375277022637751, -0.0065987544972681824, 0.028801857334943247, -0.03083430127048699, -0.006338013506044037, 0.0018853584111601486, -0.006491783714801533, -0.016058974398720793, 0.003192406466167442, -0.012682712045707952, -0.02134065200998329, 0.01335796470257504, 0.02444280190754833, -0.00974101878750419, -0.006147471887086811, -0.004763538611285445, -0.015002639435261853, -0.01097118138888676, 0.027224040389036017, -0.01486892572434789, 0.002473697132650096, 0.02874837334069382, -0.03177029352159432, 0.023961434783432305, 0.006090643513382247, 0.007588233070931442, -0.01086421060642011, 0.030914529124506325, 0.002752823828974107, -0.0015552535604562425, -0.008196628872341549, -0.020404659758875954, 0.020859284885867267, -0.008377142475207657, 0.010596784115914786, 0.025779935291397545, -0.020899398812876935, 0.010904324533429778, -0.014802068868890909, 0.029096026753895884, -0.0127295119376601, -0.03340159818704138, 0.016633941306741108, -0.007220521297240646, -0.02159470750192626, -0.011947288498326361, 0.01814490232851396, -0.020845912955982312, 0.034016680884716564, 0.008530912218303854, 0.007895774419769033, 0.014240472959432949, 0.009453534402171431, -0.005839931003910519, -0.007929202381836224, 0.003984658154422946, 0.016914740658453987, -0.016259544033769134, 0.011218550915887249, 0.0030871071749364117, -0.003610260648620323, -0.002411854750899974, 0.0059001018944250215, 0.0005908459001357144, -0.0014792040852799793, -0.018586157388265512, -0.05019599706446636, 0.02442943184030857, -0.0037807453040727143, -0.01743622170957968, 0.010135473023980997, -0.011659805044316204, 0.019254724080190123, -0.032385376219269506, 0.004318941500385251, -3.692786875848215e-05, -0.011933917499764005, 0.00970090392917192, 0.011512720334839886, -0.007340863543930952, -0.03118195654633425, -0.0015652820422086598, 0.02688975704307371, 0.011853689645744668, 0.007808860135145919, 0.025432281878195486, 0.0052047927397144, -0.007668461390612079, 0.02382772107251834, -0.020163974334172743, -0.009761075751009023, 0.004118371399675609, -0.006013758641834149, -0.002781238015826389, -0.007233892761464303, -0.015310179852776844, 0.006772581669530513, -0.041156971068308776, -0.007354235008154608, 0.028828601194713158, 0.004733453166028193, -0.007862345526379244, -0.007855659561436766, -0.0026692529924835307, 0.031074984832545, 0.022303388120860537, -0.020979626666896272, -0.014989267505376895, -0.007133607943940131, -0.015831662766547735, 0.005559133049181539, -0.005880044930920187, -0.02464337340524187, 0.006144128904615572, 0.01367219108503251, -0.008337028548197989, 0.019802948991085728, 0.020083746480153406, -0.014267215887880262, 0.00042119707761262645, -0.026488617772977025, -0.025325310164406235, 0.009099194092704292, 0.018719871099179476, 0.004238713646365914, 0.001241027177998772, 0.008738167818294675, -0.006505155179025189, 0.007146978942502487, 0.011238607879392083, 0.021327281942743533, -0.00217952771369746, 0.014507899449938274, -0.0020875998866900863, 0.002164484991068834, -0.007929202381836224, -0.024268974269624696, -0.009179421946723628, 0.008604455038703312, 0.01568457712574882, -0.0003094210779857145, 0.003530032794600986, -0.01704845436936795, 0.001293676823614287, -0.016179316179749798, 0.025713079367263164, 0.024870685037414925, -0.011813575718735, -0.021180196301944614, 0.026181076424139433, -4.967242554335056e-05, 0.03348182604106072, 0.010576727152409952, 0.0072472642256879586, 0.001216791602820214, 0.014467785522928605, -0.005274992111981319, -0.009400049476599406, 0.001656374240013549, 0.009908160460485342, 0.004676624792323629, 0.02259755753981317, -0.008337028548197989, -0.009640733038657418, 0.008637883000770502, 0.029363454175723808, -0.012194658956649451, 0.0015494035739622243, 0.00854428414818881, -0.02382772107251834, -0.01750307949635926, -0.0033796051026534287, -0.022022590631792856, -0.010857524641477632, -0.018372215823332212, 0.0032007636895148893, 0.022891726958765807, 0.006428269841815791, 0.005438790802491233, 0.01631302989066376, -0.009159364983218794, 0.0141335021769663, 0.0010346070907951107, -0.008163200910274358, -0.024162004418480647, 0.02939019617284852, 0.028908829048732498, 0.020992998596781227, 0.011820260752354877, -0.018586157388265512, 0.015350293779786514, 0.005047679548485664, 0.006231042723577387, -0.0347387315708906, 0.011272036772781874, 0.025753193294272836, 0.020912768880116693, -0.023279496161622736, -0.02632816020229315, -0.01803793247736991, -0.010717125896943791, -0.009219535873733297, 0.021768535139849887, 0.02885534319183787, -0.014080017251394275, 0.05016925320469645, 0.011786832790287687, -0.012214715920154285, 0.0017800591199291186, 0.014400928667471623, 0.013973046468927625, -0.012000774355220987, 0.027571695664883277, -0.018011190480245198, -0.0005252427795207731, 0.007748689244631415, -0.02635490406206306, 0.026782787191929662, -0.03217143651698141, -0.008243428764293695, -0.010663640971371766, -0.021354023939868245, 0.02134065200998329, 0.0043690839091473374, 0.0026792814742359478, 0.016112460255615416, 0.004432597782133079, 0.004161828309156516, -0.010382842550981486, -0.025846793078177128, -0.01814490232851396, -0.015002639435261853, -0.002909937020202842, -0.010028502241514347, -0.016794398877424982, 0.0019137724815971055, 0.0024034977603831762, -0.02361377950758504, -0.005846616503191697, 0.017235652074531337, -0.0011022995754545765, 0.00414511432812292, -0.013331221774127728, 0.009995073348124557, 0.01307716628218476, -0.01670079909352069, 0.02194236277777352, -0.005983672730915598, -0.03963264184194137, 0.031422638245747064, 0.008477427292731828, -0.02000351862613407, 9.678548927152591e-05, -0.0028297089333528553]' \n", - "LIMIT 3\n" - ] + "data": { + "text/plain": [ + "AIMessage(content='The songs about breakouts obtained from the top 5 albums about love are \\'Royal Orleans\\', \"Nobody\\'s Fault But Mine\", \\'Achilles Last Stand\\', \\'For Your Life\\', and \\'Hots On For Nowhere\\'.')" + ] + }, + "execution_count": 49, + "metadata": {}, + "output_type": "execute_result" } ], "source": [ - "import re\n", - "\n", - "def replace_brackets(match):\n", - " words_inside_brackets = match.group(1).split(', ')\n", - " embedded_words = [str(embeddings_model.embed_query(word)) for word in words_inside_brackets]\n", - " return \"'\" + \"', '\".join(embedded_words) + \"'\"\n", - "\n", - "updated_sql = re.sub(r'\\[([\\w\\s,]+)\\]', replace_brackets, query)\n", - "print(updated_sql)" + "full_chain.invoke({\"question\": \"I want to know songs about breakouts obtained from top 5 albums about love\"})" ] }, { @@ -758,20 +626,13 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Let's handle everything using LCEL" + "This is something **different** that **couldn't be achieved** using standard metadata filtering over a vectordb." ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] } ], "metadata": { "kernelspec": { - "display_name": "auto-gpt", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -786,8 +647,7 @@ "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.18" - }, - "orig_nbformat": 4 + } }, "nbformat": 4, "nbformat_minor": 2