Alexandre Lacoste: On the Path of Foundation Models for EO: Progress, Bottlenecks, and Possibilities

So,

our next speaker is Alexandre Lacoste, while he is setting up his computer.

Let

me introduce him. Alexandre is a top research manager leading the UI assist team at ServiceNow.

Since

his PhD in theoretical machine learning,

he

has published influential works on foundation models for earth observation, sequential decision-making, causality, and various works in collaboration with Yoshua Bengio.

Prior

to ServiceNow, he was the first research scientist to join Element AI and worked for three years at Google.

He

is also a founding member of Climate Change AI and an author of the seminal paper Tackling Climate Change with Machine Learning.

It's

our pleasure to host here, Alexandre.

Thank

you so much for the introduction. I'm very grateful to be here, surrounded by passionate people about a topic that's pretty close to my heart. Yeah, today I'm going to, it's going to be more like a high-level talk, trying to see where we can go with foundation model and with already little bottlenecks on the road. But first, I'd like to ask myself this grounding question, what if we can build a model that reasons about the earth the same way that GPT can think about the internet? But perhaps more precisely, why did earth observation didn't get this GPT moment yet? Is it because it's a much harder problem? Is it there's not enough data out there to be able to serve us?

Or are we just lacking incentive to get together and build this bigger kind of model? Or is it currently happening? We just don't know

yet.

But first, what do we mean by foundation model for monitoring? Which kind of capability are we expecting that it's going to have? Of course, I think we are all expecting there's going to be some kind of good model, multi-sensory fusion, where you can leverage multispectral data, maybe hyperspectral data, some wind fields, elevation map, or temporal sensing. Infuse all of that together just, you know, going beyond just the RGB optical signal that we're used to. Multi-temporal is also going to be an important part, being able to understand the different timestamps.

of

the satellite or UAVs are revisiting the same region often across time. And the temporal difference is not always the same. Sometimes it's a couple of days apart, and sometimes it can be a season apart. And, yeah, we're getting used these days to model. We can talk with. We're,

you

know, being able to express what we want, what we're searching for, like we saw in the previous demo. I think it can be very exciting. But something that's underexplored here that I'd like to talk a little bit is very large context window for monitoring. We've seen this happening in the large language model. The context window has been increasing, unleashing very interesting application. But here we're still focusing on segmentation of small images in many cases. Of course, we'd like to have reasoning in the model being able to produce a report on what it observed.

And also being able to communicate on the image that you provided, being able to communicate polygons to detect object, but it's not necessarily obvious, because it has to reason back into the original image space. Temporal reasoning and forecasting. We can picture, you know, forecasting, the risk of forest fires in some region, and having calibrated forecasts, something very, very underexplored at the moment. And also pixel-wise output, you know, being able to express the model, express itself through pixel by pixel, to be able to produce masks and some kind of overlay you can put on your image.

Yeah, some kind of any-to-any model, but perhaps with a little bit more focus on what matters for monitoring. But in between the downstream applications and the foundation model, there's going to be a list of a layer of tools. I'd just like to glance over that with you rapidly. We're getting used to chatting with our models, being able to put documents or image there. So obviously, like, we would like to have some kind of chat interface where we can put your data set and perhaps search for a specific thing a little bit like

similar

to the demo that we've seen, but a lot more interactive,

and

even if we have those zero-shot model that we can interact with, we're still going to need fine-tuning. So the ecosystem of fine-tuning models, having a pool of open-source model that we can fine-tune with, and having a bunch of fine-tuning tools in order to serve downstream application more efficiently, as it's going to be necessary. But something that is just starting this year, the agentic stack of EO, and some of

the

things that I'm excited about, imagine being able to do deep research over all of your data and being able to create a report on the fly about a topic of your interest. But

so

it's your agent to be able to go back and forth and search information, and perhaps it can identify uncertainty in some area, and realize, okay, what if I could have more data here, and if there's an API for communicating with satellite image provider, perhaps you can start asking, okay, are you going to have a satellite going over there, and then negotiate price maybe across different satellite provider, and then get more data, and then do the analysis again, produce an updated report. All of that can go, could,

you

know, improve some existing application.

Great.

But there's no reason to do any of this if there's no downstream application. One application that's going to see a lot of improvements with this usage of zero shot models and increased capability is going to be disaster response. So you have newly acquired image of a disaster that just happened, and you want to find, I don't know, what are the damaged buildings to be able to access rapidly. Very important one. Agriculture and food security is definitely going to be served. There's a lot of

on

-tap progress that can happen, both on the farmer's side and on the regulation side. The farmer can improve the precision agriculture, being able to forecast the health of your plants, forecast the yield. On the regulation side, perhaps we can uniformize a little bit and have a better understanding of what's happening with farmers and it creates everything for everyone. A topic that's dear to my heart is the carbon biomass accounting in order to improve a carbon credit system

that

can be a little bit more stable, more trustable. Of course, city planning and infrastructure monitoring

on

the elephant in the room. Any progress we're going to do is also going to have benefits for military application.

Great.

So if we step back, on one side you have capabilities, and then in between you're

going to

have layer of tools that we're going to build together, and

most

importantly, we have to ground ourselves in downstream application. So that's why we're going to jump into benchmarking.

So

if done correctly, benchmarking can help the community move together. We can guide new directions. We can open up new challenges, and we can also improve the quality of science. In many cases, you know, papers are going to develop their new technique, and then they're going to choose some way of evaluating a couple of datasets out there that

are

going to show that it improves compared to a baseline. While this works for publishing, it's not necessarily comparable from one paper to the other. It makes it, you know, can give an illusion of progress.

But

out there, there's more than 10,000 EO datasets. This is definitely enough for doing the benchmarking that we need. But how are we going to select the right subset in order to be able to know what we want? So we're going to start by narrating a couple of desiderata that we would like to have from benchmarking. Ideally, we would be able to evaluate more than one capability. So if the goal is to evaluate a foundation model, to compare it to other, there's different capabilities that the model can have. We want each capability daily on more than a few datasets. So we can have aggregated metrics, have some statistical significance.

And these are challenging tasks where there's some progress remain to be done, and we're not already plateaued. Leaderboard can help follow the progress.

Okay.

Let's dive into that.

So

throughout the talk, I collected a few high picture like this. These are done through deep research. Of course, they tend to hallucinate a little bit. They make mistakes. But I did multiple deep research, cross-validated then, and a little bit untuned, all of that. So apologies if you've been working on some of these projects and it's badly categorized. But I think it's mostly correct for these kind of table.

So

these are different benchmarks for fine-tuning. So we're still in the fine-tuning regime. So you get a large model that's been pre-trained on some data, and you would like to evaluate how well does it do if I would fine-tune it on my downstream dataset. All of these benchmarks correspond to a collection of datasets that are going to evaluate a few capabilities. Some of them are more specific. For example, FOMO bench is about forestry, and REO bench is more about robustness to noise. So the synthetically induced noise, realistic noise on images, some kind of glitch you can have from satellite noise, and the major robustness to it.

But then by looking at the table like that, we can see other axes, the verticals here. I find it very interesting, where on the side, do we see the mouse here? Yes, we do. So these two columns here have almost no representation. So this notion of evaluating for large contexts or for forecasting,

not

really out there in the field. So, you know, perhaps we should start looking at it. So temporal, like most benchmarks, don't consider temporality here. So some of them do, starting to do it. Often it's just temporal difference. But I think there's a lot of new things we can do by working on temporality.

I

should definitely put that more out here. Another thing here, there's a comment about open license. Just a note, I'm going to come back to it later if I have time. All of these benchmarks are open license, but some of them have commercial restrictions. And this

means

that if you're a company, for example, like Google, there could be legal action if you're just evaluating on that benchmark. So it makes it hard, like we need to make effort to have free for non-commercial models.

Okay.

So now we're going to dive into GeoBench. This is one of the first project I did when I entered the field of earth monitoring. And a little bit of a newcomer out here is going to be GeoBench 2. I was hoping to say it's out, but it's not out yet. It's just a few weeks of work. The GitHub is there, but the paper is going to be there in a few weeks.

Okay.

But let's talk about it. What is GeoBench 2? It's going to be 20 datasets. They're down-sampled from the original datasets, your existing dataset.

So

they're down-sampled to be able to have a faster evaluation. They are all selected to be discriminative and challenging, meaning that we experimented with more than 20. I think it was 30, 35 datasets that we all converted and trained 15 different models. And the ones that were non-discriminative, meaning that it's kind of plateauing or it's hard to really tell if one model is better than the other. Then we just discarded them.

We

also broke the benchmark in 10 different subsets. Here it makes a table. We see the different datasets that we transformed. And all of these check marks here as representing, is it part of each of the subsets? So these subsets here,

they

represent capabilities that we're trying to evaluate. And so each of these capabilities, they contain multiple datasets. So the positive side of that is if you care about temporality, you don't have to evaluate on all the 20 datasets. You can only focus on four. And you're still going to have a leaderboard that you're going to compare to other models that perhaps evaluate on all the datasets. All fully open source license. It was not necessarily easy. We had to drop very interesting datasets because they had the dash NC associated label.

And

we evaluated on more than 15 different models.

Okay.

So that was it for the fine tuning approach. But there's the zero shot model that are becoming more and more interesting for the fields. It's a very different way of evaluating this model. We don't necessarily... We don't have to fine tune. So the evaluation part is much easier in that sense.

And

there's already a couple of benchmarks that are out there. But as we see for the verticals here, there's

still

a variety of capabilities that are underexplored. Luckily, multi-sensor is there up to a certain extent. But temporality is very ignored still and large context and forecasting is not there either. And also, it seems like our community is not a big fan of leaderboards. These can be convenient

and

help the community move forward. So

this

year with MBZ UAI and IBM, we worked on GeoBench VLM.

It's different

than GeoBench. This is meant to evaluate the VLM, the zero shot model. So we're going to dive into that one really quick. So it's meant for zero shot evaluation. There's

eight

categories of tasks in that benchmark. For example, temporal understanding. An example is count the total number of damage building visible in that scene. So the model is going to have to look at both images and be able to understand which buildings is damaged. And it's multiple choices to leave no ambiguity for the evaluation. There's also object counting and localization

and

non-optical imagery. And the whole benchmark is over 10,000 instants. They are all human verified. And they're non-ambiguous evaluation.

And

also, there's a provided leaderboard.

There

are still limitations. There's only two steps for temporality. And there's no large context window over the evaluations. There's no forecasting. Perhaps it's going to come later in our next version. Now I'm going to move on to bottlenecks.

So across

the working on different projects, the

it

was a few obvious, but

when

you start working on it, but at the high level, it's not always obvious.

And

I'm going to just describe a few of them.

So

if you start collecting image for self supervision,

they

come out there uniformly across the planet. There's a lot of amounts, a large amount of data. That's just, you know, not interesting per se. There's no pattern. I mean, there's some amount of pattern, but it's mostly noise. It can be desert, ice sheets or forest. Also here, we see well enough. They, this is Landsat, the size of one single pixel. So you miss all of these detail here, Sentinel-2. Even though it's very low resolution at 10 meter, it's still very large rectangle. And so these are all open access. But if you want to go

to

better resolution, then you have to go to get that commercial data.

So

it's, it's hard to get something that are going to gather all the patterns that you care about on the earth. It's for a project we did in collaboration with Satellogic. We collected data of 15 terapixels that contains multi temporal data. This was the aim was to, to, to build a large data sets, not have a strong self supervision signal. So Satellogic was kind enough to give a

creative

common image without the non-commercial limitation. And this is the part where it's a one meter resolution and there's also near infrared. So it's at least one

band

beyond RGB.

Um,

also we collected the Sentinel-2 and Sentinel-1 image. They're all collected across the world and together so that we,

you

can, uh, inform your Sentinel-2 from Sentinel-1 and vice versa. There's also a multi temporal, uh, there's up to 10 times steps, uh, for each location. So they are, um, they're kind of cues collected together in there with overlapping. So you can create some self supervision signal. Uh, we also collected hyperspectral data,

um,

very interesting because it's not, uh, often that you can get your hand on such data.

Um,

still open source data, but very limited coverage, unfortunately.

Uh,

now bottleneck two that have been hinting about is, uh, non-commercial. This, um, uh, if, if we can, you know, make a little effort to get rid of those, uh, that would help.

Uh,

they help both on the training side and, uh, on the benchmarking side. Uh, so if, if we collect large amount of data, then commercial company, uh, if there's a dash NC, can it really use it? Or maybe they will use it anyway and they will develop a big model, but they will have to hide, um, which data they use, uh, for training it.

A

little bit, um,

uh,

like speculation about LAMA. Uh, it's been trained on a lot of,

uh,

copyrighted data, but they don't mention what they train on because,

you

know, it's like, it's gonna be legal action against them. Uh, so it makes less transparency on the training process and so on. So benchmarking,

um,

so if

you

make a beautiful benchmark out there, uh, and, and there's some data set that has dash NC. And if you're a commercial company, you cannot evaluate on it because you can have legal action. So it's just slowing down everyone. So,

uh,

if we can, uh, just focus on,

on

the one, no dash NC. Um, so recommendation, if you build a new data set, uh, you can try to make an effort that, you know, you're partnering with people, um,

uh,

partnering with the provider. And so you can make an effort to negotiate, um, to remove this dash NC. Sometimes it's not gonna be, uh, possible.

And

sometimes there can be, uh, some compromise you can make, uh, by maybe lowering the resolution of the data.

Uh,

they're gonna be more inclined to release it.

Uh,

you can try to get a grant or partnering with high profile, um, uh, association to create the beautiful story behind it. Also important, communicate the license and the process clearly. There's, you know, if, if there's a data set, there's gonna be true component to the license. There's gonna be the code component, often there's code associated. Um, and then there's the, the, the labels that you create has their own license. And then the image itself may have a different license. It's important to commit the, the tree license, communicate where it comes from and how you got it. Uh, because we've been in a situation where legal department is looking into that.

It's like, yeah, it says it's CC by, um, but I'm not sure it's not clearly identified. And, and sometimes people, they just stamp without, um, knowing what they're doing. So leave a little bit of traces, put more than just, this is the license.

Um,

this is gonna help. Okay.

Uh,

bottleneck four.

Uh,

the cost of satellite imagery. I was, I was really interested. Where are we heading with that? If we're gonna build a large model, then, um, are we gonna need to collect a lot of data? And especially the high resolution data, what's been limiting us in the high resolution and multi-sensor is the cost of gathering all of that. Um, so I,

I,

you know, I collected a little bit of data, uh, put that plot here. And I was curious about the, some kind of projection.

Um,

you know, like

these

are calculated number with just a few, uh, things to take out of that with a grain of salt. But we see a trend where, um, airborne, um, or UAV imagery is going down much faster than, uh, satellite imagery,

uh,

which satellite imagery have six year

timeframe

for halving the cost. Um, so I was, I just broke down the cost to try to understand where, what is the bottleneck. Um, there's a lot of costs for satellite imagery that is, um, so by the way, this is a cost for satellite, not for, uh,

drones.

Um, a lot of it's a big part of it is computer related, which is somewhat of a good news. Meaning it's going to go down a little faster. Uh, the processing, the storage, uh, the data transfer, the, transferring to the ground station,

um, count

for about 30%, 35%. Um, the satellites, uh, part of the actual satellite that you have to build, especially if you want to have a high resolution satellite,

you

need the, the lens with that, the optics. Uh, it's very, um,

uh,

heavy and expensive stuff. Um, it doesn't follow Moore's law. It's, it's more traditional, uh, engineering. So the costs to half, uh, the time to half costs is more on the scale of five to 10 years. It's really one of the bottleneck. Um, and one of the reasons I was interested in two dads, because we know the cost of launching stuff into space is going down very fast, but it turns out it's not a big part of the cost. And of course, it's other people, uh, other costs like people and the market.

Uh, uh,

but perhaps there's gonna be a little bit plot twist to this, um,

uh,

cost forecast. Maybe the agentic stack if we're able to have, um,

uh,

agent negotiating prices and lowering the friction, uh, to get those satellite images. Uh, maybe it's gonna amortize costs a little faster and we're gonna be seeing a different dynamic. Or maybe I'm just

too

optimistic.

Uh,

now, um, I'm gonna dive down into some potential research access.

Uh,

where, uh, where I believe we should put some, uh, some effort into, uh, improving, uh, the, um, our field.

Uh,

we saw in the previous talk,

uh,

a mascot to encoder. It's very famous one. Um, it's, um, it's,

um, it's

an elegant solution to sensor fusion

or

even temporal fusion. Uh, what you do, you patchify everything and you put some encoding,

uh,

associated to each patch. Uh, this encoding is gonna encode the temporal,

uh,

the spectral information, maybe the sensor information,

and,

and also the spatial information. So, uh, well, in our case, it's starting to be a lot of information to pack in this encoding and associated to the patch. But the,

the

advantage of that is that once this patch is identified like that, you can ensemble it into a set of, uh, sparse elements of your image. So this means that you need, uh, all the sensor. You can, sometime you have SAR, sometime you have,

uh,

optical, sometime you have elevation, or sometime you just have a part of the image on one side and then

a

part of the other side. This is what allows you to train in a mascot to encoder fashion, where you just mask some of the inputs and then try to predict the output. It's this notion of sparse set,

uh,

that allows us to do that. So many papers, uh, I've been working with this, uh, principle,

uh,

and it works to some extent, but there's a, um, a core limitation to this approach is you're losing important fine grain detail.

Uh,

the, for example, here, if your job is to, uh, identify cattle, uh, in the field, this is a project I was working on.

Uh,

we're trying to count, uh, cows in the, uh, Amazon rainforest to track

illegal

deforestation.

But

if you patchify your image 16 by 16 or even 8 by 8, you're losing this core information. I mean, you may be able to recover a little bit of it, but it's like your, your small element detail is going to be clumped together.

Uh,

so we see there's a clear trade-off. So VIT and the mascot encoder are good for sensor fusion, but, uh, can get fine-grained details.

On

the other hand, um, you can play with ConvNex and SWIN architecture. These are very good at fine-grained details. We observed very good performance on some, um, data sets and, uh, both GeoBench 1 and GeoBench 2. Um, these are the two architecture that stands out

compared

to most of VIT architecture.

Um,

the problem is that you're back to this dense D-dimensional tensor.

You

cannot break it down to in sparse something. And so, yeah, we're back to square zero in terms of, uh, sensor fusion and the scalability of training this whole thing. Uh, so how, what can we do? Would there be some kind of hybrid approach where we can combine both? Um, not sure. Do we need to develop new architecture in order to be able to move forward?

Um,

you tell me. Just open research.

Uh,

research access number two. I've been talking about very long, uh, spatial and temporal window. Uh, I was curious to see if there was a trend. So this is yet another deep research. Uh, I tried to get the, uh, receptive fields of

many

of the so-called foundation model, uh, that are out there.

Um,

receptive fields is

kind

of, um, you know, there's some, some things are patchified. So it's the number of token, the size of the token can vary. So I tried to get a metric that's, um, come into all of them. So it's a

kilo

pixels.

So

with the, the, the,

the

y-axis that we see here is the number of kilo pixels of the receptive field.

Um,

so

one

thing that's interesting is there's absolutely no trend. It's more like everyone is working under, um, some 3,000,

uh,

kilo pixels of receptive field.

And

what I mean by pixel RGB is three pixels here. So there's multiple bands, uh, or there's multiple temporality. These, these all add up.

Uh,

so it depends on, on, I'm calculating, multiplying this, this kind of cube.

Uh,

there's three outliers, uh, but there's two of them that are, um, at least when glancing at the paper, it seems more like a rapid decrease of, you know, exponential aggregation, a different layer. So it's

kind

of release data is like a

big

window because you're not integrating information across all of this thing. So SkySense was one that seemed to be a little bit more, uh, involved into that. Um, so, but what, what are the research access then? Um, maybe first question, should we even go larger? Does it even make sense? Am I, am I just like wishful thinking or translating from the LLM world where we clearly see a trend increasing? Um, which kind, uh, if,

if

we do, which kind of, uh, dataset would require such thing? I, I'm thinking that, uh, yield forecasting or the,

the,

some kind of agriculture where we get, um, weekly screenshot across the year and you have the whole crop. Yeah. I can integrate a lot of information if you ask specific, um,

uh,

difficult question where the information isn't there. So perhaps integrating all of that information would, would allow you to, um,

uh, uh,

respond interestingly.

So,

but we need to build this dataset. If they are out there, please let me know. Um, and, and yeah, of course the,

um,

obvious research in that directions, like explore different architecture and training method. Um, you kind of have to skate it up. This is the kind of, uh, research that require a lot of GPUs.

Now, um,

as a third research access, and there's a lot of work that need to be done on, um, um,

uh,

VLM as well. Uh, we've seen rapid progress this year. This is another, uh, table that shows, um,

uh,

different

VLM

for, uh, EO reasoning.

Uh,

what we see is

it's

still, um, largely around language and object detection or object, uh, image captioning.

There's

very little multi-sensor, um, multi, very little multi-temporal.

On

the benchmarking side, um, we saw

that

it's ready, but for, on the training, the,

um,

there's little work on that.

Uh,

perhaps because there's still a lot of progress to be done just on, on these things, or perhaps just because of data, um, like George was communicating with us. Like it's, you know, it's hard to get this

text

and multi-spectral, uh, match together. Same thing with Tempolity.

Uh,

but yeah, we should, we should try to collect these new benchmark, this new dataset.

Great.

Um, now let's talk about research access number four.

Uh,

the agentic stack. So there's already, this is very new field. The, um, um,

uh,

when I started this presentation, um, I thought it was a little bit forcing to start talking about agents. But I, and there's actually, uh, quite a bit of agents out there already, and there's already to benchmark.

Uh,

my other agenda is about, um,

uh,

agents,

um,

more like LNM agents,

um,

able to interact in user and,

um,

like computer use agent or web agent.

Um,

so I, I'm, I'm excited to see that happening in the EU world.

Um,

so if we go back, if we focus on the benchmark side,

uh,

there's two, uh, two benchmarks here. One done by MBZUAI. Um, the, this 436, uh, human curated queries. And they are close to the VLM kind of queries, I would say.

But

they are interesting because they're complex where,

uh,

it's probably a good idea for your VLM to break it down into different steps. For example, you have to calculate, um, uh, something, you first have to detect some element in the image. And then you have to do some calculation around it. So it's like,

you

can, you have a sandbox where you can run code and then come back to it and then detect something else on the image. Technically a VLM could do it in one shot without doing tool calling.

But,

um,

uh,

an agent perspective can help you improve, uh, the score.

On

the other hand, you have Univ Earth, um, Google Earth Engine as a tool. And, uh, the, it's about researching information and then bringing it back.

Now, on

the agent side, there's already a variety of agents. I'm gonna see some why, uh, they are coming out all over the place. Uh, one particular one is, um,

uh,

Google Geospatial Reasoning. Project, the, I only saw a demo. It's not usable yet. But you see that chat window where you can, um, start querying on, um,

uh,

Google Map. And, and have like Jim and I interacting with, uh, with your data and search into database. Uh, very impressive demo. I encourage you to go see it.

But

what are the, um, challenges ahead of us, um, for that? So

what's

interesting is that agents and tools, it can be coded in a few days. Or, um, even in the afternoon if you vibe code it.

Um,

you know, MCP protocol allows you to, um, create very easy tools around maybe QGIS. Or maybe you wrap up, uh, some Earth model that you have. Uh, Earth observation model. You just wrap it up as a tool. And then, then you can have, um, GPT or Claude. And they're gonna call, start calling them as tools. And they can start interacting. Um, very cool things. So,

making

agents is not so hard. Um, but making good benchmark takes months. And, and then this, this subfield is gonna, is gonna branch out also into so many different subfields. It's gonna be like, hey, if we can make an agent for this, we can also make an agent for that. Uh, and so there's many startup ideas or many, um, you know, like this, this notion of being able to query, uh, images, uh, from satellite provider through an API, uh, sounds very appealing to me. Um, but we're gonna clearly need a benchmark. Cause I don't wanna release, uh, an agent out there that's gonna buy a million dollars of, um, remote sensing imagery, uh, for no reason. You wanna be able to calibrate it and,

you know,

be able to make sure it's doing the right thing.

Uh,

it's also a little bit hard to frame as a research direction. Like making a benchmark.

You

bring it as a, as a research paper. It's accepted and it's a lot of work, but it's accepted.

And

then making an agent that does something,

um,

and trying to make it small bench. I don't, it's harder to have an accepted paper. Anyway, so this, this still a research axis.

Um,

it's hard to, to do a research on agent. Um, great. So I went a little faster than I was expecting.

Um,

and I, so I can answer some questions. There's time over. I'm gonna go back to slides and I hide because I thought I was having too many slides.

All

right. Thank you very much. Uh, are there questions?

Thank

you for the interesting talk. I have a question about agents. Namely what you mentioned at the end that, uh, agents and tools can be called very quickly. Um, perhaps that's true if we think of agents like, uh, some pre-trained model that we just feed some stuff in context. But don't you think that they, like, if we want to progress there, we want to actually look into datasets and training them with reinforcement learning to actually be able to solve truly complex tasks.

I

completely agree with you on that. Um, and yes. The,

uh,

you have to, you can create environments, a synthetic environment where, uh, you know, perhaps a fake environment where you can do this and then you can start to doing reinforcement learning to improve this.

Um,

work, having worked on that for a couple of years on the UI agent side, um,

it's,

it's, it works.

It's

very slow progress. And then, you know, you get this, this new agents that you got, you show that Lama,

your

open source model can make that progress. And then you get this new closed source model that comes out and out of the box, it gets really, uh,

um,

um, uh, good performance. Um, but yeah, they, so the two world exists. So that I don't definitely don't want to discourage that. And it's definitely a really good access of research. So yeah, creating, creating synthetic environment where you can do reinforcement learning, uh, for your agent sounds very, very appealing project for me.

Thank you.

Uh,

thank you for the very interesting presentation. Uh, my question is more related to the data fusion side of, uh, of it.

Um,

our brain collects, uh, data from our surroundings using multiple senses,

but

we rely more on some than on others when there's conflicting information.

With

more and more satellites and, uh, eyes in the sky,

do

you think there will be, do you see some similar trends on what type of

data

and satellite, uh, images we can rely more on?

Hmm.

Hmm. You mean, well, they, because you, I mean, I don't know how to answer that one, but there's availability on one side and then there's, uh, being able to rely on it. Uh, there's some downstream application that would be really benefit from having different kinds of data.

Uh,

perhaps,

uh,

something that can penetrate the ground of the cold data is not able to penetrate the ground. But, uh, some bands or some radar is able to go analyze soil.

But

if we want to make progress in there, we're going to not need to start collecting data sets for that.

Yeah.

The physics of, uh, capturing different data satellite images is different and we somehow know some of it.

But,

uh, how much we can rely on different tasks to deal with this data and to make decisions based on

multiple

data sources.

Hmm.

That's what I was alluding to.

How

much can you say, are you meaning that since there's a lot of the, the spatial data is not so available, it's not going to be as reliable?

Or, uh,

it's more than, uh, whether you think that some types of, uh, of data, uh, based on the physical properties of how they are captured. Mm

-hmm. And

what they mean

will

be more reliant upon, uh, when making decisions because

we

see more and more multi-sensor data areas of interest. Yeah.

Yes. So, I, I do think if we make progress on, on sensor fusion, then, uh, perhaps we're going to be able to, uh, perceive more directly

the

information that we care about instead of inferring it from patterns.

That,

that, that could definitely help some reliability.

Thank

you.

Are

there more questions?

Um,

I have one, uh, by the way, in the meantime. Uh, you talked about, uh, this, uh, advantage and disadvantage of transformers and then CNN,

uh,

architectures, especially compared to,

uh, swing

against the other VITs, right? And then, um, the pros and cons about, um, that one is good for diffusion, the transformers, and now there's more about the fine grain details. I didn't quite understand what makes to do so, although you commented about this

block

tensors, not having

these

sparse things, but it's still not clear on what particular application we're talking about.

Uh,

what are the difficulties?

And

then maybe you can give some more details on that.

Yes. Um, so you,

you

want to know why exactly, um,

uh,

so, so there's, on, on one side, you have the, the SWIN and, uh, the CONVNEXT

that

are

working

directly with this dense tensor thing and it's able to have a more conventional approach.

And it's,

we see in, in, in

metrics

when we measure on benchmark

compared

to VIT, the performance is a little higher.

And

the hypothesis is that it's able to, um, uh, make better usage of this fine grain information that we see in the image. Because we're often limited by, um, the resolution. We wish we would have more pixels, but we, we don't have it. Like 0.3 meter,

uh,

is the best we can get.

And

we, we need to make the best we can out of these, um,

uh,

these pixels. Um, so in CONVNEXT

seems

to be one of the, you know, it's, it's more con, uh, conventional, convolutional architecture. Uh, on the other side, if you do VIT style, you're going to have to patchify it.

Uh,

some people reduce the size of patches. I think it can work. Um, so we're going to have maybe four by four patches. So then you increase your granularity, uh, like that. I, I haven't made explicit experiments to understand that.

Um,

but if you keep it to eight by eight or 16 by 16, you're kind of, you know, if you're, you're in your patch,

you

have two, three cow

in

it. You won't be able to distinguish. It's going to be like a linear prediction or, uh, some kind of blob embedding of, of different components. So yeah, you lose, um, very important information if you're going to patchify, uh, your image.

So,

but when you patchify it like that and you, you bring the encoding, uh, spectral temporal and all of that with your patch, then you, you don't have, uh, this dense

image

anymore, this dense tensor,

um,

and, and you can have a sparse set of components.

And

this is how transformer works. It works with a set of components. And this sparsity allows us to, uh, turn off some sensors. So sometime you only have RGB. Sometime you only have, uh, SAR. So we have different input and we can have two inputs at the same time or three inputs. You can have multiple time steps. So it, it lets you being more flexible on, on, on what you put in there.

So

when

the

fusion we're talking about is 3D diverse kinds of sensors. Okay. And also about this, uh,

um,

the prediction task. Because I was thinking about if you are doing this dense prediction task of, let's say, fine-grained semantic segmentations, then transformers also do quite well, right?

And, uh,

there it, it doesn't seem to have a problem of,

you

know, patchifications because they also have to still make a predictions

at

the pixel level, right? And then is it like,

then,

then is it intrinsically different in these settings we're talking about in the geospatial settings? Or, um, it's still unclear why it works with a similar logic that you, that you, that you describe on email segmentation, fine-grained email segmentation,

transfers only that problem. So you're

saying that transformer do, uh, express some fine-grained, um,

uh,

details. It, so it can work up to, uh, to some level of fine-grained.

So

I don't have a clear answer there, you know, like it's kind of the limit of, um, my, um, what I distilled from experience. Um, but I, um, so I, I think it's, it's able to have some, uh, uh, understand some subspace that's going to be able to be precise at the segmentation level, but it's going to be limited, uh, to

that. I think,

I think my question is more about, is it, like, generic problem or are we talking about more specific to geospatial data that has different problems in this?

I,

I,

I think, I think it's more common in geospatial data that you're limited to your resolution, right? When you, you have common image from the web, uh, it's often high resolution and you have, like, high level semantics, uh, object that you care about. Uh, but I think if you, even from normal image, if you start caring about,

uh,

many very specific details,

then,

then this patch level, uh, is going to hurt you a little bit more. And this is something we see a lot in maybe one third of geospatial application where you're, if you could have better pixels, less blurry, or high resolution, then it's going to be better. And this is where this, this patch thing is, is, it's hurting a little bit.

All

right. Thank you very much. Are

there

any more questions? If there are no more questions, then, uh, let's thank the speaker again for the amazing talk. Uh, we, this, uh, now we have, uh,

coffee

break, uh, right outside, uh,

um,

you get, uh, you can grab the coffee. But in the meantime, I also want to update you regarding the posters. Uh, there will be posters, uh,

uh,

hanging there.

Not

necessarily being presented now.

If

that's big, uh, if they are,

um,

the authors around,

you

know, you may face them, discuss about, hang out during the coffee breaks. Uh, those posters will be rather more better presented tomorrow. Tomorrow is, uh, more spotlight talks related to those posters as well as the poster presentations,

uh,

will be more active during the breaks. Uh, today also feel free to go over and reach out the, the, the,

the

authors and discuss about the posters.

All

right. Uh, see you in half an hour.