-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
277 lines (248 loc) · 21.2 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
<!DOCTYPE HTML>
<html>
<head>
<title>Robot Manipulation with Flow Matching</title>
<meta charset="utf-8" />
<meta name="viewport" content="width=1000">
<link rel="stylesheet" href="assets/css/main.css" />
<link rel="icon" type="image/x-icon" href="images/favicon.ico">
<!-- Google tag (gtag.js) -->
<script async src="https://www.googletagmanager.com/gtag/js?id=G-9YQ4D13KL7"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'G-9YQ4D13KL7');
</script>
<!-- please do not copy the google tag, as the 'https://fastumi.com/' did -->
<meta property="og:url" content="flow-matching-policy.github.io" />
<meta property="og:type" content="website" />
<meta property="og:title" content="Affordance-based Robot Manipulation with Flow Matching" />
<meta property="og:description" content="We present a framework for assistive robot manipulation, which focuses on two fundamental challenges: first, efficiently adapting large-scale models to downstream scene affordance understanding tasks, especially in daily living scenarios where gathering multi-task data involving humans requires strenuous effort; second, effectively learning robot trajectories by grounding the visual affordance model. We tackle the first challenge by employing a parameter-efficient prompt tuning method that prepends learnable text prompts to the frozen vision model to predict manipulation affordances in multi-task scenarios. Then we propose to learn robot actiontrajectories guided by affordances in a supervised Flow Matching method. Flow matching represents a robot visuomotor policy as a conditional process of flowing random waypoints to desired robot trajectories. Finally, we introduce a real-world dataset with 10 tasks across Activities of Daily Living to test our framework. Our extensive evaluation highlights that the proposed prompt tuning method for learning manipulation affordance with language prompter achieves competitive performance and even outperforms other finetuning protocols across data scales, while satisfying parameter efficiency. Learning multi-task robot action trajectories with flow matching also leads to better generalization performance and faster inference in various manipulation tasks than some alternative behavior cloning methods. Our framework seamlessly unifies affordance model learning and trajectory generation with flow matching for robot manipulation." />
</head>
<body id="top">
<!-- Main -->
<div id="main" style="padding-bottom:1em; padding-top: 5em; width: 60em; max-width: 70em; margin-left: auto; margin-right: auto;">
<section id="four">
<div class="box alt" style="margin-bottom: 1em;">
<div class="row 0% uniform" style="width: 100%; display: flex; justify-content: space-between;">
<div class="2u" style="font-size: 1em; line-height: 1.5em; text-align: center; width: 20%">
<span class="image fit" style="margin-bottom: 0.5em;">
<img src="images/logo.png" alt=""/>
</span>
</div>
</div>
</div>
<h2 style="text-align: center; white-space: nowrap; color: #4e79a7; font-size: 200%"> Affordance-based Robot Manipulation <br> with Flow Matching</h2>
<span class="figure" style="margin-top: 0.5em; margin-bottom: 0;">
<img src="images/overall.png" width="100%" margin-top="0.5em" alt="" />
</span>
<p class="paper-paragraph" align="justify">
We present a framework for assistive robot manipulation, which focuses on two fundamental challenges: first, efficiently adapting large-scale models to downstream scene affordance understanding tasks, especially in daily living scenarios where gathering multi-task data involving humans requires strenuous effort; second, effectively learning robot trajectories by grounding the visual affordance model. We tackle the first challenge by employing a parameter-efficient prompt tuning method that prepends learnable text prompts to the frozen vision model to predict manipulation affordances in multi-task scenarios. Then we propose to learn robot actiontrajectories guided by affordances in a supervised Flow Matching method. Flow matching represents a robot visuomotor policy as a conditional process of flowing random waypoints to desired robot trajectories. Finally, we introduce a real-world dataset with 10 tasks across Activities of Daily Living to test our framework. Our extensive evaluation highlights that the proposed prompt tuning method for learning manipulation affordance with language prompter achieves competitive performance and even outperforms other finetuning protocols across data scales, while satisfying parameter efficiency. Learning multi-task robot action trajectories with flow matching also leads to better generalization performance and faster inference in various manipulation tasks than some alternative behavior cloning methods. Our framework seamlessly unifies affordance model learning and trajectory generation with flow matching for robot manipulation.
</p>
<hr style="margin-top: 0em;">
<h3>Highlights</h3>
<div class="box alt" style="margin-bottom: 1em;">
<div class="row 50% uniform" style="width: 100%;">
<div class="2u" style="font-size: 0.9em; line-height: 1.5em; text-align: center; width: 59.5%">
<span class="image fit" style="margin-bottom: 1.5em;">
<video autoplay loop muted playsinline style="width: 100%; margin-right: 5%;"><source src="videos/flow.mp4" type="video/mp4"> </video>
</span>
Flow matching represents a robot visuomotor policy as a conditional process of flowing random waypoints to desired robot actions.
</div>
<div class="2u" style="font-size: 0.9em; line-height: 1.5em; text-align: center; width: 40.5%">
<span class="image fit" style="margin-bottom: 1.5em;">
<video autoplay loop muted playsinline style="width: 100%; margin-right: 5%;"><source src="videos/pt.mp4" type="video/mp4"> </video>
</span>
Prompt tuning for vision-language-model to predict manipulation affordances in multi-task scenarios.
</div>
</div>
<div class="row 50% uniform" style="width: 100%;">
<div class="2u" style="font-size: 0.9em; line-height: 1.5em; text-align: center; width: 59.5%">
<span class="image fit" style="margin-bottom: 0.5em;">
<video autoplay loop muted playsinline style="width: 100%; margin-right: 5%;"><source src="videos/diffusion.mp4" type="video/mp4"> </video>
</span>
Flow matching exhibits slightly better generalization performance, noticeably faster inference and greater stability on training and evaluation than diffusion model.
</div>
<div class="2u" style="font-size: 0.9em; line-height: 1.5em; text-align: center; width: 40.5%">
<span class="image fit" style="margin-bottom: 1.5em;">
<video autoplay loop muted playsinline style="width: 100%; margin-right: 5%;"><source src="videos/feed.mp4" type="video/mp4"> </video>
</span>
An example of robot feeding the human with flow matching.
</div>
</div>
</div>
<hr style="margin-top: 0em;">
<h3>Real-world Experiments</h3>
<p>
Flow Matching has been tested on 10 tasks across Activities of Daily Living, and leads to consistently better performance than alternative behavior cloning methods. (Videos are 4x speed)
</p>
<div class="box alt" style="margin-bottom: 1em;">
<div class="row 50% uniform" style="width: 100%;">
<div class="2u" style="font-size: 1em; line-height: 1.5em; text-align: center; width: 25%">
<span class="image fit" style="margin-bottom: 0em;">
<video autoplay loop muted playsinline style="width: 100%; margin-right: 5%;"><source src="videos/comb.mp4" type="video/mp4"> </video>
</span>
Comb the hair
</div>
<div class="2u" style="font-size: 1em; line-height: 1.5em; text-align: center; width: 25%">
<span class="image fit" style="margin-bottom: 0em;">
<video autoplay loop muted playsinline style="width: 100%; margin-right: 5%;"><source src="videos/trash.mp4" type="video/mp4"> </video>
</span>
Sweep the trash
</div>
<div class="2u" style="font-size: 1em; line-height: 1.5em; text-align: center; width: 25%">
<span class="image fit" style="margin-bottom: 0em;">
<video autoplay loop muted playsinline style="width: 100%; margin-right: 5%;"><source src="videos/hang.mp4" type="video/mp4"> </video>
</span>
Hang the towel
</div>
<div class="2u" style="font-size: 1em; line-height: 1.5em; text-align: center; width: 25%">
<span class="image fit" style="margin-bottom: 0em;">
<video autoplay loop muted playsinline style="width: 100%; margin-right: 5%;"><source src="videos/bottle.mp4" type="video/mp4"> </video>
</span>
Pass the water
</div>
</div>
<div class="row 50% uniform" style="width: 100%;">
<div class="2u" style="font-size: 1em; line-height: 1.5em; text-align: center; width: 25%">
<span class="image fit" style="margin-bottom: 0em;">
<video autoplay loop muted playsinline style="width: 100%; margin-right: 5%;"><source src="videos/hat.mp4" type="video/mp4"> </video>
</span>
Put on the hat
</div>
<div class="2u" style="font-size: 1em; line-height: 1.5em; text-align: center; width: 25%">
<span class="image fit" style="margin-bottom: 0em;">
<video autoplay loop muted playsinline style="width: 100%; margin-right: 5%;"><source src="videos/arm.mp4" type="video/mp4"> </video>
</span>
Wipe the forearm
</div>
<div class="2u" style="font-size: 1em; line-height: 1.5em; text-align: center; width: 25%">
<span class="image fit" style="margin-bottom: 0em;">
<video autoplay loop muted playsinline style="width: 100%; margin-right: 5%;"><source src="videos/teeth.mp4" type="video/mp4"> </video>
</span>
Brush the teeth
</div>
<div class="2u" style="font-size: 1em; line-height: 1.5em; text-align: center; width: 25%">
<span class="image fit" style="margin-bottom: 0em;">
<video autoplay loop muted playsinline style="width: 100%; margin-right: 5%;"><source src="videos/blooper.mp4" type="video/mp4"> </video>
</span>
😠😲😊🤖🧑🦲📊🔬🏍️🦾
</div>
</div>
</div>
<hr style="margin-top: 0em;">
<h3>Franka Kitchen Simulation Experiments</h3>
<div class="box alt" style="margin-bottom: 1em;">
<div class="row 50% uniform" style="width: 100%;">
<div class="2u" style="font-size: 1em; line-height: 1.5em; text-align: center; width: 25%">
<span class="image fit" style="margin-bottom: 0em;">
<video autoplay loop muted playsinline style="width: 100%; margin-right: 5%;"><source src="videos/kitchen1.m4v" type="video/mp4"> </video>
</span>
</div>
<div class="2u" style="font-size: 1em; line-height: 1.5em; text-align: center; width: 25%">
<span class="image fit" style="margin-bottom: 0em;">
<video autoplay loop muted playsinline style="width: 100%; margin-right: 5%;"><source src="videos/kitchen3.m4v" type="video/mp4"> </video>
</span>
</div>
<div class="2u" style="font-size: 1em; line-height: 1.5em; text-align: center; width: 25%">
<span class="image fit" style="margin-bottom: 0em;">
<video autoplay loop muted playsinline style="width: 100%; margin-right: 5%;"><source src="videos/kitchen2.m4v" type="video/mp4"> </video>
</span>
</div>
<div class="2u" style="font-size: 1em; line-height: 1.5em; text-align: center; width: 25%">
<span class="image fit" style="margin-bottom: 0em;">
<video autoplay loop muted playsinline style="width: 100%; margin-right: 5%;"><source src="videos/kitchen4.m4v" type="video/mp4"> </video>
</span>
</div>
</div>
</div>
<hr style="margin-top: 0em;">
<h3><a id="paper">Paper</a></h3>
<p>
<a href="https://arxiv.org/abs/2409.01083">2409.01083 [cs.RO]</a>.<br>
Affordance-based Robot Manipulation with Flow Matching<br>
Fan Zhang, Michael Gienger<br>
<br>
Code is here <a href="https://github.com/HRI-EU/flow_matching">https://github.com/HRI-EU/flow_matching</a>.<br>
We are in process of integrating flow matching into the Hugging Face 🤗 LeRobot PushT task.
</p>
<hr style="margin-top: 0em;">
<h3>Team</h3>
<section>
<div class="box alt" style="margin-bottom: 0em;">
<div class="row 50% uniform" style="width: 100%;">
<div class="2u" style="font-size: 1em; line-height: 1.5em; text-align: center; width: 16%">
<a href="https://fan6zh.github.io/">
<span class="image fit" style="margin-bottom: 0.5em;">
<img src="images/fan.jpg" alt="" style="border-radius: 50%;" />
</span>Fan Zhang
</a>
</div>
<div class="2u" style="font-size: 1em; line-height: 1.5em; text-align: center; width: 16%">
<a href="https://www.honda-ri.de/">
<span class="image fit" style="margin-bottom: 0.5em;">
<img src="images/michael.png" alt="" style="border-radius: 50%;" />
</span>Michael Gienger
</a>
</div>
</div>
</div>
</section>
<hr style="margin-top: 0em;">
<section>
This webpage template was recycled from <a href='https://diffusion-policy.cs.columbia.edu/'>here</a>.
</section>
<svg version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" height="0" width="0" style="position: absolute;">
<defs>
<!-- Go-up -->
<g id="go-up">
<path d="M10,9V5L3,12L10,19V14.9C15,14.9 18.5,16.5 21,20C20,15 17,10 10,9Z" fill="#696969"/>
</g>
<!-- Folder -->
<g id="folder" fill-rule="nonzero" fill="none">
<path d="M285.22 37.55h-142.6L110.9 0H31.7C14.25 0 0 16.9 0 37.55v75.1h316.92V75.1c0-20.65-14.26-37.55-31.7-37.55z" fill="#FFA000"/>
<path d="M285.22 36H31.7C14.25 36 0 50.28 0 67.74v158.7c0 17.47 14.26 31.75 31.7 31.75H285.2c17.44 0 31.7-14.3 31.7-31.75V67.75c0-17.47-14.26-31.75-31.7-31.75z" fill="#FFCA28"/>
</g>
<g id="folder-shortcut" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
<g id="folder-shortcut-group" fill-rule="nonzero">
<g id="folder-shortcut-shape">
<path d="M285.224876,37.5486902 L142.612438,37.5486902 L110.920785,0 L31.6916529,0 C14.2612438,0 0,16.8969106 0,37.5486902 L0,112.646071 L316.916529,112.646071 L316.916529,75.0973805 C316.916529,54.4456008 302.655285,37.5486902 285.224876,37.5486902 Z" id="Shape" fill="#FFA000"></path>
<path d="M285.224876,36 L31.6916529,36 C14.2612438,36 0,50.2838568 0,67.7419039 L0,226.451424 C0,243.909471 14.2612438,258.193328 31.6916529,258.193328 L285.224876,258.193328 C302.655285,258.193328 316.916529,243.909471 316.916529,226.451424 L316.916529,67.7419039 C316.916529,50.2838568 302.655285,36 285.224876,36 Z" id="Shape" fill="#FFCA28"></path>
</g>
<path d="M126.154134,250.559184 C126.850974,251.883673 127.300549,253.006122 127.772602,254.106122 C128.469442,255.206122 128.919016,256.104082 129.638335,257.002041 C130.559962,258.326531 131.728855,259 133.100057,259 C134.493737,259 135.415364,258.55102 136.112204,257.67551 C136.809044,257.002041 137.258619,255.902041 137.258619,254.577551 C137.258619,253.904082 137.258619,252.804082 137.033832,251.457143 C136.786566,249.908163 136.561779,249.032653 136.561779,248.583673 C136.089726,242.814286 135.864939,237.920408 135.864939,233.273469 C135.864939,225.057143 136.786566,217.514286 138.180246,210.846939 C139.798713,204.202041 141.889234,198.634694 144.429328,193.763265 C147.216689,188.869388 150.678411,184.873469 154.836973,181.326531 C158.995535,177.779592 163.626149,174.883673 168.481552,172.661224 C173.336954,170.438776 179.113983,168.665306 185.587852,167.340816 C192.061722,166.218367 198.760378,165.342857 205.481514,164.669388 C212.18017,164.220408 219.598146,163.995918 228.162535,163.995918 L246.055591,163.995918 L246.055591,195.514286 C246.055591,197.736735 246.752431,199.510204 248.370899,201.059184 C250.214153,202.608163 252.079886,203.506122 254.372715,203.506122 C256.463236,203.506122 258.531277,202.608163 260.172223,201.059184 L326.102289,137.797959 C327.720757,136.24898 328.642384,134.47551 328.642384,132.253061 C328.642384,130.030612 327.720757,128.257143 326.102289,126.708163 L260.172223,63.4469388 C258.553756,61.8979592 256.463236,61 254.395194,61 C252.079886,61 250.236632,61.8979592 248.393377,63.4469388 C246.77491,64.9959184 246.07807,66.7693878 246.07807,68.9918367 L246.07807,100.510204 L228.162535,100.510204 C166.863084,100.510204 129.166282,117.167347 115.274437,150.459184 C110.666301,161.54898 108.350993,175.310204 108.350993,191.742857 C108.350993,205.279592 113.903236,223.912245 124.760454,247.438776 C125.00772,248.112245 125.457294,249.010204 126.154134,250.559184 Z" id="Shape" fill="#FFFFFF" transform="translate(218.496689, 160.000000) scale(-1, 1) translate(-218.496689, -160.000000) "></path>
</g>
</g>
<!-- File -->
<g id="file" stroke="#000" stroke-width="25" fill="#FFF" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round">
<path d="M13 24.12v274.76c0 6.16 5.87 11.12 13.17 11.12H239c7.3 0 13.17-4.96 13.17-11.12V136.15S132.6 13 128.37 13H26.17C18.87 13 13 17.96 13 24.12z"/>
<path d="M129.37 13L129 113.9c0 10.58 7.26 19.1 16.27 19.1H249L129.37 13z"/>
</g>
<g id="file-shortcut" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
<g id="file-shortcut-group" transform="translate(13.000000, 13.000000)">
<g id="file-shortcut-shape" stroke="#000000" stroke-width="25" fill="#FFFFFF" stroke-linecap="round" stroke-linejoin="round">
<path d="M0,11.1214886 L0,285.878477 C0,292.039924 5.87498876,296.999983 13.1728373,296.999983 L225.997983,296.999983 C233.295974,296.999983 239.17082,292.039942 239.17082,285.878477 L239.17082,123.145388 C239.17082,123.145388 119.58541,2.84217094e-14 115.369423,2.84217094e-14 L13.1728576,2.84217094e-14 C5.87500907,-1.71479982e-05 0,4.96022995 0,11.1214886 Z" id="rect1171"></path>
<path d="M116.37005,0 L116,100.904964 C116,111.483663 123.258008,120 132.273377,120 L236,120 L116.37005,0 L116.37005,0 Z" id="rect1794"></path>
</g>
<path d="M47.803141,294.093878 C48.4999811,295.177551 48.9495553,296.095918 49.4216083,296.995918 C50.1184484,297.895918 50.5680227,298.630612 51.2873415,299.365306 C52.2089688,300.44898 53.3778619,301 54.7490634,301 C56.1427436,301 57.0643709,300.632653 57.761211,299.916327 C58.4580511,299.365306 58.9076254,298.465306 58.9076254,297.381633 C58.9076254,296.830612 58.9076254,295.930612 58.6828382,294.828571 C58.4355724,293.561224 58.2107852,292.844898 58.2107852,292.477551 C57.7387323,287.757143 57.5139451,283.753061 57.5139451,279.95102 C57.5139451,273.228571 58.4355724,267.057143 59.8292526,261.602041 C61.44772,256.165306 63.5382403,251.610204 66.0783349,247.62449 C68.8656954,243.620408 72.3274172,240.35102 76.4859792,237.44898 C80.6445412,234.546939 85.2751561,232.177551 90.1305582,230.359184 C94.9859603,228.540816 100.76299,227.089796 107.236859,226.006122 C113.710728,225.087755 120.409385,224.371429 127.13052,223.820408 C133.829177,223.453061 141.247152,223.269388 149.811542,223.269388 L167.704598,223.269388 L167.704598,249.057143 C167.704598,250.87551 168.401438,252.326531 170.019905,253.593878 C171.86316,254.861224 173.728893,255.595918 176.021722,255.595918 C178.112242,255.595918 180.180284,254.861224 181.82123,253.593878 L247.751296,201.834694 C249.369763,200.567347 250.291391,199.116327 250.291391,197.297959 C250.291391,195.479592 249.369763,194.028571 247.751296,192.761224 L181.82123,141.002041 C180.202763,139.734694 178.112242,139 176.044201,139 C173.728893,139 171.885639,139.734694 170.042384,141.002041 C168.423917,142.269388 167.727077,143.720408 167.727077,145.538776 L167.727077,171.326531 L149.811542,171.326531 C88.5120908,171.326531 50.8152886,184.955102 36.9234437,212.193878 C32.3153075,221.267347 30,232.526531 30,245.971429 C30,257.046939 35.5522422,272.291837 46.4094607,291.540816 C46.6567266,292.091837 47.1063009,292.826531 47.803141,294.093878 Z" id="Shape-Copy" fill="#000000" fill-rule="nonzero" transform="translate(140.145695, 220.000000) scale(-1, 1) translate(-140.145695, -220.000000) "></path>
</g>
</g>
</defs>
</svg>
<!-- Footer -->
<!-- <footer id="footer">
<div class="inner">
<ul class="copyright">
<li>Meet <a href="https://en.wikipedia.org/wiki/Danbo_(character)">Danbo</a> the cardboard robot.</li>
</ul>
</div>
</footer> -->
<!-- Scripts -->
<script src="assets/js/jquery.min.js"></script>
<script src="assets/js/jquery.poptrox.min.js"></script>
<script src="assets/js/skel.min.js"></script>
<script src="assets/js/util.js"></script>
<!--[if lte IE 8]><script src="assets/js/ie/respond.min.js"></script><![endif]-->
<script src="assets/js/main.js"></script>
</body>
</html>