Items tagged with: filter
Advanced ffmpeg techniques
Converting part of a video into a high quality gif with 256 colors per frame
The one liner:
ffmpeg -ss 61.0 -t 2.5 -i video.mp4 -filter_complex "[0:v] fps=6,scale=w=700:h=-1,split [a][b];[a] palettegen=stats_mode=single [p];[b][p] paletteuse=new=1" photo.gif
Let's break it down shall we?
This is pretty obvious, but if you are new to #ffmpeg, this is the call to launch the #executable.
This tells the ffmpeg program to fast seek 61.0 seconds into the #video (start here please).
From 61.0 seconds into the video, we only want to create a #gif from the next 2.5 seconds of video.
The "-i" is required for all ffmpeg #conversion that specifies what file will will converting from.
This is where things start to get weird, but I promise that it is worth it!
Use the first #stream in the video (typically the video stream, as videos normally have two streams or more), use 6 frames per second to create the gif, set the width to be 700 pixels and the height to be automatically determined.
From our #input, we are making two outputs, one named "a" and the other "b". You can name them whatever you want, but keep it consistant throughout the rest of the code.
;[a] palettegen=stats_mode=single [p]
The semicolon tells the program that we are going to pass it another #filter. #Palettegen generates one palette for a whole video stream. Stats_mode=single says for each frame, compute a new #histogram.
The palletteuse=new=1 compliments the previous code and indicates to the filter that for each frame it should grab a new pallete.
The name of the file that you are creating.
Now that you have all this advanced knowledge of ffmpeg, you are probably wondering, "Why couldn't I just use
ffmpeg -i video.mp4 photo.gif?" Well, this is clearly for advanced, high quality looking gif making with videos that have lots of colors. Enjoy! (source)
Ale historia mi się przydarzyła. :)
Jakiś czas temu kupiłem w sklepie internetowym filtr przepływowy do brzeczki od Bouncer. Świetne rozwiązanie na każdym etapie. Przy przelewaniu brzeczki z gara do fermentora oddziela chmieliny które przedostały się przez hopstopper, a gdyby się przytkało to jednak jest na wierzchu a nie na dnie gara.
Przy rozlewie pozwala zatrzymać chmiel z chmielenia na zimno i rozlać praktycznie 100% zawartości fermentora do butelek. Przy standardowej siatce (900 µm) czasem coś przeleci, ale co do zasady u mnie to rozwiązanie się sprawdza i filtr ratuje mi co najmniej dwie flaszki podczas rozlewu i likwiduje całkowicie problem zatykającego się zaworka w rurce do rozlewu.
No ale zdarzyło się tak, że przekonałem się, że nie każde tworzywo dobrze reaguje na StarSan.
Generalnie SS stosuję tak, że sprzęt którego mam zamiar używać leży sobie zanurzony w nim i czeka na swoją kolej, jak jest potrzebny to wyjmuję, i wiem, że nic na nim nie przeżyło. Szklanka od filtra niestety jest z takiego pechowego tworzywa. Dlaczego wypadek zdarzył się dopiero przy którymś z kolei użyciu nie wiem, ale zrobiło się tak, że cała szklanka się zmatowiła, i popękała. Wygląda jak gdyby ją walnąć młotkiem na tyle mocno, że pojawiły się pęknięcia ale jednak się nie rozpadła.
Dziwne, ale ponieważ StarSan to produkt od jankesów i tam popularny, filtr takoż napisałem maila opisując sytuację i pytając co może być przyczyną (przychodziło mi do głowy, że może zareagował SS z desprayem który używałem w tej samej okolicy). odpowiedzieli, że tak, wiedzą, sugerują nie moczyć w SS długo tylko przepłukać przed użyciem, a jak podam adres to wyślą mi zamiennik. Adres podałem oczywiście i otrzymałem odpowiedź:
Hi Jakub,WOW. No to czekam. Paczka dotarła do .pl 29.04 i wylądowała na cle. Długi weekend więc spokojnie czekam rozumiejąc, że może się przeciągnąć, ale po dwóch tygodniach zadzwoniłem zapytać co tam. Okazało się, że wysłali mi zawiadomienie 30.04 … nic nie mam więc musi im zginęło… tia :D (wczoraj przyszło jakieś awizo to pewnie to, dwa tygodnie z jednego urzędu pocztowego do drugiego na odległość ok 10km… idą na rekord :D).
I am sending you a package with a replacement bowl, and several other things that I think you will enjoy
We are gratified and humbled that you are finding our products useful, so far away from us.
Dziś rano (w nocy kurka, wstałem o 6:00 jak zwierze dzikie :D), poszedłem odebrać. Okazało się, że nie ma to tamto muszę pomóc w zasypaniu słynnej dziury w VAT i nawet jeśli nie zapłaciłem to paczka ma zadeklarowaną wartość 30USD, co pan skrzętnie przeliczył na 207zł (drogie te dulary :D) od czego jest 48zł VAT plus jakaś opłata za przelew (16 ?) plus opłata za wydanie… 8. Taaaa :D
No ale to nieważne, odebrałem, poszedłem do fabryki, otwarłem i… szczęka spadła mi na podłogę. Minęło już trochę czasu, ale nadal jej do końca nie pozbierałem.
I jedyne co mnie martwi to, że z jednej strony chciałbym powiedzieć wszystkim (także szerzej niż tutaj w serwisie dla socjalnych anarchistów :D) ale boję się, że jakiś łoś pomyśli, że to super pomysł na interes. Wydawało by się, że już jesteśmy cywilizowani, ale całkiem niedawno netflix dał znać, że jednak nie. :(
#HomeBrewing #Bouncer #filter
I have to tell you about the Kalman filter, because what it does is pretty damn amazing. Surprisingly few software engineers and scientists seem to know about it, and that makes me sad because it is…
HN link: https://news.ycombinator.com/item?id=17116149
Posted by sytelus (karma: 7814)
Post stats: Points: 145 - Comments: 12 - 2018-05-21T04:32:14Z
#HackerNews #2015 #filter #how #kalman #pictures #works
HackerNewsBot debug: Calculated post rank: 100 - Loop: 246 - Rank min: 100 - Author rank: 43
HN Discussion: https://news.ycombinator.com/item?id=19666148
Posted by dessant (karma: 702)
Post stats: Points: 167 - Comments: 68 - 2019-04-15T15:47:08Z
#HackerNews #adblock #arbitrary #code #execute #filter #lists #may #pages #plus #web
HackerNewsBot debug: Calculated post rank: 134 - Loop: 220 - Rank min: 100 - Author rank: 51
wisst ihr was mir gerade einfällt, wer richtig in Trouble geraten wird? Wikia, denn die leben zu einem ganz erheblichen Teil von Urheberrechtsverletzungen und verdienen damit auch ihr Geld (nach Definition der geplanten Richtlinie).
Ich meine… guckt euch doch nur mal die ganzen Fandom-Wikis an wie Zeldapedia oder das Harry Potter -Lexikon oder The Walking Dead (TV) Wiki.
Ich glaube nicht, dass Wikia die ganzen Bilder (und teilweise Videos), die die Autoren da so hochladen lizensiert hat und ich glaube auch nicht dass die Autoren das getan haben.
Aber künftig wird Wikia das kontrollieren müssen. Und ich kann mir gut vorstellen, dass diese Fandom-Wikis um einiges ärmer würden ohne ihre Bebilderung. Es mag allerdings denkbar sein, dass die meisten (vielleicht nicht alle) Bilder als Zitat klar gehen, soweit sie eben in einen größeren Kontext (eben den Wiki-Artikel) eingebettet sind.
Jetzt gehört #Wikia zugegebenermaßen nicht zu meinen Lieblings-Firmen (aber es gibt Schlimmere XD), aber trotzdem ist das ein Bereich in dem es echt zu Schwierigkeiten kommen wird denke ich.
Aber ich glaube auch: Ein Media-Wiki selbst zu betreiben ist nicht viel aufwändiger als ein Wordpress ;D
#wiki #fandom #bilder #filter
The development could help provide clean drinking water for millions of people who lack access to safe sources.
Article word count: 891
HN Discussion: https://news.ycombinator.com/item?id=19315295
Posted by 11thEarlOfMar (karma: 4866)
Post stats: Points: 140 - Comments: 47 - 2019-03-05T23:46:40Z
#HackerNews #drinking #filter #for #graphene-based #purifies #salt #water
By Paul Rincon Science editor, BBC News website
Graphene oxide membrane Image copyright Uni Manchester Image caption Artwork: Graphene-based membranes hold huge promise in desalination
A UK-based team of researchers has created a graphene-based sieve capable of removing salt from seawater.
The sought-after development could aid the millions of people without ready access to clean drinking water.
The promising graphene oxide sieve could be highly efficient at filtering salts, and will now be tested against existing desalination membranes.
It has previously been difficult to manufacture graphene-based barriers on an industrial scale.
Reporting their results in the journal Nature Nanotechnology, scientists from the University of Manchester, led by Dr Rahul Nair, show how they solved some of the challenges by using a chemical derivative called graphene oxide.
Isolated and characterised by a University of Manchester-led team in 2004, graphene comprises a single layer of carbon atoms arranged in a hexagonal lattice. Its unusual properties, such as extraordinary tensile strength and electrical conductivity, have earmarked it as one of the most promising materials for future applications.
But it has been difficult to produce large quantities of single-layer graphene using existing methods, such as chemical vapour deposition (CVD). Current production routes are also quite costly.
On the other hand, said Dr Nair, "graphene oxide can be produced by simple oxidation in the lab".
He told BBC News: "As an ink or solution, we can compose it on a substrate or porous material. Then we can use it as a membrane.
"In terms of scalability and the cost of the material, graphene oxide has a potential advantage over single-layered graphene."
Image caption Access to clean water remains a major issue for millions of people around the world
Of the single-layer graphene he added: "To make it permeable, you need to drill small holes in the membrane. But if the hole size is larger than one nanometre, the salts go through that hole. You have to make a membrane with a very uniform less-than-one-nanometre hole size to make it useful for desalination. It is a really challenging job."
Graphene oxide membranes have already proven their worth in sieving out small nanoparticles, organic molecules and even large salts. But until now, they couldnʼt be used to filter out common salts, which require even smaller sieves.
Previous work had shown that graphene oxide membranes became slightly swollen when immersed in water, allowing smaller salts to flow through the pores along with water molecules.
Now, Dr Nair and colleagues demonstrated that placing walls made of epoxy resin (a substance used in coatings and glues) on either side of the graphene oxide membrane was sufficient to stop the expansion.
More about graphene
What is graphene?
Graphene patent surge reveals global race
How sticky tape trick led to Nobel Prize
How could graphene transform the future?
Is graphene really a wonder material?
Restricting the swelling in this way also allowed the scientists to tune the properties of the membrane, letting through less or more common salt for example.
When common salts are dissolved in water, they always form a "shell" of water molecules around the salt molecules.
This allows the tiny capillaries of the graphene-oxide membranes to block the salt from flowing through along with the water.
"Water molecules can go through individually, but sodium chloride cannot. It always needs the help of the water molecules. The size of the shell of water around the salt is larger than the channel size, so it cannot go through," said Dr Nair.
Image copyright PHOTOSTOCK-ISRAEL/SCIENCE PHOTO LIBRARY
Image caption The scientists plan to test the graphene oxide sieve against existing industrial membranes used in desalination
By contrast, water molecules flow exceptionally fast through the membrane barrier, which makes it ideal for use in desalination.
"When the capillary size is around one nanometre, which is very close to the size of the water molecule, those molecules form a nice interconnected arrangement like a train," Dr Nair explained.
"That makes the movement of water faster: if you push harder on one side, the molecules all move on the other side because of the hydrogen bonds between them. You can only get that situation if the channel size is very small."
By 2025 the UN expects that 14% of the worldʼs population will encounter water scarcity. As the effects of climate change continue to reduce urban water supplies, wealthy modern countries are also investing in desalination technologies.
Current desalination plants around the world use polymer-based membranes.
"This is our first demonstration that we can control the spacing [of pores in the membrane] and that we can do desalination, which was not possible before. The next step is to compare this with the state-of-the-art material available on the market," said Dr Nair.
In a news and views article accompanying the study in Nature Nanotechnology, Ram Devanathan, from the Pacific Northwest National Laboratory in Richland, US, said more work needed to be done to produce graphene oxide membranes inexpensively at industrial scales.
He added that scientists also needed to demonstrate the durability of the membranes under prolonged contact with seawater and ensure the membrane was resistant to "fouling" by salts and biological material (which requires existing barriers to be periodically cleaned or replaced).
"The selective separation of water molecules from ions by physical restriction of interlayer spacing opens the door to the synthesis of inexpensive membranes for desalination," wrote Dr Devanathan.
"The ultimate goal is to create a filtration device that will produce potable water from seawater or wastewater with minimal energy input."
Follow Paul on Twitter.
HackerNewsBot debug: Calculated post rank: 109 - Loop: 162 - Rank min: 100 - Author rank: 37
As we've been discussing the "Trilogue" negotiations between the EU Commission, EU Council and EU Parliament over the EU's Copyright Directive have continued, and a summary has been released on the…
Article word count: 2671
HN Discussion: https://news.ycombinator.com/item?id=18652378
Posted by em3rgent0rdr (karma: 3682)
Post stats: Points: 141 - Comments: 88 - 2018-12-11T00:30:21Z
\#HackerNews #block #but #copyright #dont #everything #filter #make #mistakes #never #upload #use
As weʼve been discussing the "Trilogue" negotiations between the EU Commission, EU Council and EU Parliament over the EUʼs Copyright Directive have continued, and a summary has been released on the latest plans for Article 13, which is the provision that will make upload filters mandatory, while (and this is the fun part) insisting that it doesnʼt make upload filters mandatory. Then, to make things even more fun, another document on the actual text suggests the way to deal with this is to create a better euphemism for filters.
When we last checked in on this, we noted that the legacy film and television industry associations were freaking out that Article 13 might include some safe harbors for internet platforms, and were asking the negotiators to either drop those protections for platforms, or to leave them out of Article 13 altogether and only have it apply to music.
The latest brief description of the recommendations for Article 13 appear to be an attempt by bureaucrats who have no understanding of the nuances of this issue to appease both the legacy copyright industries and the tech companies. Notably absent: any concern for the public or independent creators. Weʼll dig in in a moment, but frankly, given the state of Article 13 demonstrated in this two-page document, it is horrific that these discussions are considered almost concluded. It is obvious that the vast majority of people working on this have no idea what theyʼre talking about, and are pushing incredibly vague rules without any understanding of their impact. And rather than taking in the criticism and warning from knowledgeable experts, theyʼre just adding in duct-taped "but this wonʼt do x" for every complaint where people warn what the actual impact of the rules will be for the internet.
Thatʼs why, throughout this document, they keep insisting that there will be no mandate for filters. But, thereʼs no way you can actually avoid liability without filters. Indeed, in order to appease the film and TV folks, the proposal now includes a notice-and-staydown provision. Weʼve spent years explaining why a notice-and-staydown provision is not only unworkable, but would lead to tremendous amounts of non-infringing content being removed. Copyright is extremely context specific. The exact same content may be infringing in one instance, but protected in another. Yet a notice-and-staydown does not allow the protected versions. It requires they be blocked. That is outright censorship.
On to the document. It starts with seven "guidelines."
The Commission was requested to follow these guidelines indicated by the Rapporteur: \* Platforms should follow high standards of duty of care; \* Cooperation should not be unidirectional; \* Non infringing content should remain on the platform online; \* Automatic blocking, albeit non forbidden, should be avoided as much as possible \* Existing measures should not be excluded \* Platforms should not always be released from liability by merely applying content identification measures \* Rightholders should not be in a worse position than they are currently. In this context the audiovisual sector was singled out. On this basis, the following ideas, which are based on a logical grouping of the above guidelines, are outlined for the consideration of the co-legislators:
This can be summed up as... all infringing content must disappear, but you donʼt have to use filters and you must make sure that non-infringing content remains online. This is the "nerd harder" approach to regulating. It is magic wand regulating: make the bad stuff go away, and magically donʼt have any collateral damage.
This is not sound policy making. This is technically illiterate bureaucrats trying to cover their asses. Because the liability requirements in the document will certainly lead to massive overblocking and widespread unintended consequences, these spineless technocrats are trying to avoid that by just tacking on "but let those consequences happen." They donʼt explain how this is possible. They just are going to put these rules out into the world, and tell the tech industry to wave a magic wand and make one type of content disappear without impacting other content (even if theyʼre impossible to distinguish, and if the penalties for getting it wrong are dire).
From there, the document provides "details" never apparently recognizing just how contradictory the plans are:
High standard of duty of care and bilateral cooperation (1 + 2) Online content sharing service providers, as defined in the directive, are considered to communicate to the public and as such need to obtain licences from the relevant rightholders. Where no licences are granted, online content sharing service providers and rightholders should cooperate in good faith to prevent the availability of protected content online. Cooperation should take place according to appropriate standards of professional diligence, which ought to take into account the size of the service, the number and type of works or other subject matter uploaded by users, the potential economic harm caused to rightholders, the availability of suitable and effective technologies and their cost for service providers. In practice, this means that the standards of cooperation should be particularly high for high value content. Cooperation should not lead to a general monitoring obligation as defined under the e-Commerce Directive.
Magic wand thinking: Either your entire platform needs to be licensed (i.e., no user-generated content) or you need to "prevent the availability" of any copyright-covered content with "good faith." But how? Well, the bureaucrats insist that this shouldnʼt require "general monitoring" (i.e., an upload filter). But... um... how do you prevent availability of copyright covered content if youʼre not monitoring? This is an impossible situation and either the bureaucrats know this and are just ignoring that theyʼre demanding the impossible, or they donʼt understand this and shouldnʼt be allowed within 10 miles of any regulation over the internet.
Rightholders should provide content sharing service providers with specific information (e.g. metadata) allowing identification of their content. The cooperation could include content identification measures (e.g. for high value content) but should not prevent other forms of cooperation if agreed by the parties (e.g. ex post content moderation for low value content, see also letter B). When unauthorised content becomes available on their websites, content sharing service providers would in general not be liable if they have cooperated in good faith according to the relevant standards of professional diligence. However, within an adequate framework to ensure legal certainty, when despite such cooperation the availability of content online has caused significant economic harm to rightholders the Directive could consider the provider liable in any event, but at a reduced level taking into account the good faith of the provider. Alternatively, the Directive could allow rightholders to claim restitution of the benefits appropriated by the providers (e.g. using unjust enrichment claims under national law) (see point C below).
So, again, we see the general incomprehensibility of what is being pushed here. The first paragraph is an attempt to appease the platforms, basically saying "if copyright holders are going to demand takedowns, they should at least be required to supply the details of what content they actually hold a copyright over." Thatʼs reasonable given a plan to demand mandatory filters, because the only thing such metadata is actually useful for is... a filter.
The second paragraph is basically saying "okay, yes, we mean filters for loosely defined ʼhigh valueʼ content, but maybe loosely defined ʼlow value contentʼ doesnʼt require filters." Again, this appears to be an attempt to split the baby. Who the hell is going to self-describe their own content as "low value content?" The whole concept of "high value" and "low value" is elitist claptrap from the legacy content industries who basically believe that anything that comes from the legacy recording, TV and film studios is "high value" and all that independent, amateur, and user-generated content is "low value." The paragraph here is supposed to be an attempt to say "well, okay, if your platform is just publishing garbage memes and stuff maybe it doesnʼt need a filter, but if you happen to include any of Hollywoodʼs precious brilliance, you must put in place a filter."
The third paragraph is, yet again, an attempt to give special extra rights to the legacy recording, TV, and film companies. It basically says that if platforms try to "cooperate in good faith" (i.e., censor at the drop of a hat) then maybe they would be considered not liable... but only if itʼs that riff-raff low value content that slips through the filters (though weʼre not demanding filters!). If any content slips through the filters that "caused significant economic harm" (i.e., comes from the big copyright industries), well then, it doesnʼt fucking matter how much you tried to stop it, youʼre still liable.
In other words, if any internet platform makes a single mistake with Hollywoodʼs content, no matter how hard they tried to stop it, too bad, youʼre liable.
And this is where thereʼs such a massive disconnect between the framers (and supporters) of Article 13 and reality. When youʼre told that any mistake will lead to liability, you are put in a position of trying to prevent any mistakes. And the only ways to do that are to (1) stop accepting any user-uploaded content or (2) filter the hell out of all of it, and take down anything that even might possibly be considered infringing, meaning tons of perfectly legitimate content will get shut down.
No matter how many times these technocrats say "donʼt take down non-infringing works", itʼs totally meaningless if the only way to avoid liability is to take down tons of non-infringing works. Which brings us to the next part:
Non infringing content should remain online and automatic blocking to be avoided as much as possible (3+4) Content that does not infringe copyright, for example because it is covered by exceptions, should stay on the services’ websites. In addition, the co-legislators could provide that minor uses of content by amateur uploaders should not be automatically blocked (in the context of the cooperation and professional diligence referred to under A) nor trigger the liability of the uploader. This should be without prejudice to the remedies under point C and the rules on liability of the providers and cooperation under A. The need to allow legitimate content to remain available, should be strengthened through a robust redress mechanism which should ensure that users can contest measures taken against their legitimate uploads. The Commission already provided possible suggestions to the co-legislators which are currently under discussions in the trilogue process.
Again, this is setting up a laughable impossibility. First they say youʼre liable if you let anything through, and then they say "but donʼt accidentally take down stuff you shouldnʼt." How the hell do you do that? The rules donʼt say. Hollywood and Article 13ʼs supporters donʼt care. Itʼs great if they add a "redress mechanism" for bogus takedowns, but that only will apply to content that first gets up and then is taken down. It says nothing for content that is blocked from being uploaded in the first place due to overaggressive filters, which are only overaggressive due to the earlier parts of Article 13 that say youʼre liable if you let anything "high value" through.
This is the ultimate in cowardice from the EU regulators. Rather than address the actual problems that their own regulations will create, these regulators have decided to just append a bit to their regulation that says "and donʼt let this create the problems it will obviously create." Thatʼs fucking useless.
Rightholders should keep benefiting from existing measures; and platforms not released from liability by merely applying content identification technologies. Rightholders, notably audiovisual sector, not worse off (5+6+7) Rightholders should in any event retain the ability to request removal of infringing content from the websites of the content sharing services. Building on and complementing the current ecommerce rules, rightholders should be allowed to request that unauthorised content is expeditiously removed and that best efforts are made to ensure that it stays down. As indicated in A, the co-legislators may provide for an additional safeguard for rightholders when despite the good faith cooperation the availability of content online causes significant economic harm to them.
Thereʼs something really big hidden in here. A "notice and stay down" requirement. That was not what was being pushed before. Notice and staydown creates all sorts of problems, in that by its very nature it obliterates the points in the previous paragraph. If you have a notice and staydown regime, you cannot allow content that is "covered by exceptions" because youʼve already designated all such content must stay down. And unless these bureaucrats in Brussels have magically invented a filter that can understand context and correctly judge whether or not something is covered by an exception (something that normally takes a years-long adversarial judicial process) it is difficult to see how this is possible.
Then we get to the other document, leaked earlier today by Politico, that attempts to wordsmith the actual language of Article 13. Itʼs basically the same stuff we discussed above, but with an attempt to put it into actual legalese. Two things stand out in the document. First, they try to rebrand mandatory upload filters, by now discussing "suitable and effective technologies" to "ensure the non-availability on the websites of the service providers of unauthorised works or other subject matter..." How is that not a filter?
This document also includes some language "as an option" that would require "best effort to prevent their future availability." Thatʼs putting the notice-and-staydown into the law. I will note that there is no real language being discussed that explains how to prevent the blocking of non-infringing works. Just more hand waving and magical thinking about how it shouldnʼt block non-infringing works... even though it absolutely will.
This leaves me with two key takeaways:
1. The bureaucrats putting this together are doing the worst kind of regulating. They appear to be utterly ignorant of what it is that they are regulating, how it works, and the inevitable impact of their new rules. And, rather than trying to take the time to actually understand the concerns, they are simply writing "but donʼt do that" into the law every time someone explains the impact. But you canʼt regulate internet platforms not to overblock when everything else in your law requires them to overblock or face crippling liability. This is like a law that says "you must immediately dump out the bathwater without looking to see whatʼs in the bath... but donʼt throw out the baby with the bathwater." How do you do that? The law doesnʼt say because the regulators donʼt have the slightest clue. And they donʼt have the slightest clue because itʼs impossible. And, they donʼt seem to care about that because once they pass the law they can celebrate and the mess they create is left for the internet platforms (and the public) to deal with.
2. Given the massive changes and broad and unclear mandates being tossed around, Article 13 is nowhere near a condition which should be put into a binding regulation. Whatʼs being debated now is so unclear, so vague and such a mess, that it would be practically criminal to put such nonsense into law. They are rushing to get this done (perhaps before the next EU Parliamentary elections next spring), and the fact that theyʼre about to make massive changes to a fundamental part of society (the internet) without clearly comprehending what theyʼre doing is incredibly frightening. This is like a bad first draft of a bad proposal. This is not just "this is a bad bill that went through a comprehensive process and I disagree with it." This is an utter mess. It keeps shifting, it has vague and contradictory definitions, it tells companies to wave magic wands, and tells companies not to let the very thing the law compels actually happen. This is not regulating. This is why the public hates regulators.
Iʼm still hopeful that common sense eventually shows up in the EU, but at this point the only way for common sense to survive is to simply dump Article 13 entirely.
HackerNewsBot debug: Calculated post rank: 123 - Loop: 169 - Rank min: 100 - Author rank: 21
Does Google show you different search results based on your personal info, even when signed out and in so-called "incognito" mode? We ran a study to find out.
HN Discussion: https://news.ycombinator.com/item?id=18608306
Posted by The_Reto (karma: 90)
Post stats: Points: 174 - Comments: 61 - 2018-12-05T14:14:02Z
\#HackerNews #bubble #click #filter #google #how #influencing #measuring #the #what #you
Over the years, there has been considerable discussion of Googleʼs "filter bubble" problem. Put simply, itʼs the manipulation of your search results based on your personal data. In practice this means links are moved up or down or added to your Google search results, necessitating the filtering of other search results altogether. These editorialized results are informed by the personal information Google has on you (like your search, browsing, and purchase history), and puts you in a bubble based on what Googleʼs algorithms think youʼre most likely to click on.
The filter bubble is particularly pernicious when searching for political topics. Thatʼs because undecided and inquisitive voters turn to search engines to conduct basic research on candidates and issues in the critical time when they are forming their opinions on them. If they’re getting information that is swayed to one side because of their personal filter bubbles, then this can have a significant effect on political outcomes in aggregate.
Back in 2012 we ran a study showing Googleʼs filter bubble may have significantly influenced the 2012 U.S. Presidential election by inserting tens of millions of more links for Obama than for Romney in the run-up to that election. Our research inspired an independent study by the Wall Street Journal (paywall):
A Wall Street Journal examination found that the search engine often customizes the results of people who have recently searched for "Obama"—but not those who have recently searched for "Romney."
Now, after the 2016 U.S. Presidential election and other recent elections, there is justified new interest in examining the ways people can be influenced politically online. In that context, we conducted another study to examine the state of Googleʼs filter bubble problem in 2018.
Google has claimed to have taken steps to reduce its filter bubble problem, but our latest research reveals a very different story. Based on a study of individuals entering identical search terms at the same time, we found that:
1. Most participants saw results unique to them. These discrepancies could not be explained by changes in location, time, by being logged in to Google, or by Google testing algorithm changes to a small subset of users.
2. On the first page of search results, Google included links for some participants that it did not include for others, even when logged out and in private browsing mode.
3. Results within the news and videos infoboxes also varied significantly. Even though people searched at the same time, people were shown different sources, even after accounting for location.
4. Private browsing mode and being logged out of Google offered very little filter bubble protection. These tactics simply do not provide the anonymity most people expect. In fact, itʼs simply not possible to use Google search and avoid its filter bubble.
Visualization of three differing Google search results for the query ʼgun controlʼ.
For those interested in more details, weʼve written out everything below, as well as provided the underlying data and code. We hope this work encourages further study of this important issue.
We asked volunteers in the U.S. to search for "gun control", "immigration", and "vaccinations" (in that order) at 9pm ET on Sunday, June 24, 2018. Volunteers performed searches first in private browsing mode and logged out of Google, and then again not in private mode (i.e., in "normal" mode). We compiled 87 complete result sets — 76 on desktop and 11 on mobile. Note that we restricted the study to the U.S. because different countries have different search indexes.
During analysis of the search results, we only looked at websitesʼ top-level domains, for example www.cdc.gov/features/vaccines-travel and www.cdc.gov/vaccines/adults would both be treated as just cdc.gov.
Finding #1: Most people saw results unique to them, even when logged out and in private browsing mode.
To count variants of results, we noted the order of the major elements: the organic (regular) links, the news (Top Stories) infobox, and the videos infobox. We ignored ads, sections containing related searches, and other infoboxes. There were variations in these too, but we didnʼt consider them.
A quick note on ordering of links: You might think that as long as the same links are shown to users, the ordering of them is relatively unimportant, but thatʼs not the case. A given link gets only about half as many clicks as the link before it and twice as many clicks as the link after it. In other words, link ordering matters a lot because people click on the first link much more than the second, and so on.
The amount of variations we saw for each search term is listed below. For this part of the study, we excluded mobile results because the number of infoboxes displayed can vary significantly between mobile and desktop. Thatʼs why it says 76 participants instead of the overall total of 87. We also controlled for location (more on that below).
Private browsing mode (and logged out):
\* "gun control": 62 variations with 52/76 participants (68%) seeing unique results. \* "immigration": 57 variations with 43/76 participants (57%) seeing unique results. \* "vaccinations": 73 variations with 70/76 participants (92%) seeing unique results.
\* "gun control": 58 variations with 45/76 participants (59%) seeing unique results. \* "immigration": 59 variations with 48/76 participants (63%) seeing unique results. \* "vaccinations": 73 variations with 70/76 participants (92%) seeing unique results.
Visualization of unique search results shown for the search query ʼgun controlʼ.
With no filter bubble, one would expect to see very little variation of search result pages — nearly everyone would see the same single set of results. Thatʼs not what we found.
Instead, most people saw results unique to them. We also found about the same variation in private browsing mode and logged out of Google vs. in normal mode.
Now, some search result variation is expected due to two factors that we controlled for. First, search results can change over time, such as the inclusion of time-sensitive links. We controlled for this factor by having everyone search at the same time.
Second, search results can change by location, such as the inclusion of local news articles. We controlled for this factor by checking all links by hand for this possibility, comparing them to the city and state of the volunteer. We saw very few local links for gun control (1 organic link, 1 news infobox link) and immigration (0), though more for vaccinations (15 organic links, 4 news infobox links).
To control for these local links, we replaced all of them with the same placeholder — localdomain.com for organic links and "Local Source" for infoboxes — in all of our analysis. This adjustment means two users whose results only differed by a different local domain in the same slot would not count as different. Interestingly, this adjustment didnʼt affect overall variation significantly.
Another reason you might expect some variation is testing of the search algorithm, where you show slightly different results to different people. In that case, youʼd expect to see most people seeing the same results, with a few people seeing slight differences. What we saw, by contrast, was most people seeing different results.
Google search results typically have ten organic links. While the ordering of those links really matters (i.e. link #1 gets ~40% of clicks, link #2 ~20%, link #3 ~10% and so on), we also wanted to know how many different domains were being displayed.
With no filter bubble, one would expect to see this total to be around ten. We saw significantly more. In private browsing mode, logged out of Google, and with local domains replaced with localdomain.com, here are the totals:
\* "gun control": 19 different domains \* "immigration": 15 different domains \* "vaccinations": 22 different domains
Visualization of domains appearing in organic search results per person.
As you can see this clearly in the visualization above, some people were shown a very unusual set of results relative to the other participants, offered some domains seen by no-one else. If you were one of these people, you would have no way of knowing what youʼre missing.
We also wanted to look at variation within the news (Top Stories) and videos infoboxes. We also saw significant variation within those, even though there are only three slots available. Again, these are for private browsing mode, logged out of Google, and with local domains replaced with "Local Source".
\* "gun control": 3 variations from 5 sources, appearing for 75/76 people. The most common variation was seen by 69 people (90%). \* "immigration": 6 variations from 7 sources, appearing for 76/76 people. The most common variation was seen by 35 people (46%). \* "vaccinations": 2 variations from 3 sources, appearing for 2/76 people. Each variation was seen by one person (1%).
\* "gun control": 12 variations from 7 sources, appearing for 75/76 people. The most common variation was seen by 24 people (32%). \* "immigration": 6 variations from 6 sources, appearing for 75/76 people. The most common variation was seen by 42 people (55%). \* "vaccinations": Not shown in the search results.
As an example, the Videos infobox for the "immigration" query showed the following six variations. As with organic search results, the ordering matters here because the second and third slots get far fewer clicks.
\* Today, MSNBC, NBC News (shown to 42 participants) \* MSNBC, Today, NBC News (shown to 26 participants) \* Today, MSNBC, MSNBC (shown to 4 participants) \* MSNBC, Today, Today (shown to 1 participant) \* New York Times, CNN, MSNBC (shown to 1 participant) \* Today, MSNBC, RealClearPolitics (shown to 1 participant)
Remember, we had people search at the same time, and we changed all local-links to the be same, so this variation is not explained by time or location. And again, some people were real outliers; in fact, some didnʼt see the infoboxes at all.
Finding #4: Private browsing mode and being logged out of Google offered almost zero filter bubble protection.
Finally, we saw the variation in private browsing mode (also known as incognito mode) and logged out of Google as about the same as in normal mode. Most people expect both being logged out and going "incognito" to provide some anonymity. Unfortunately, this is a common misconception as websites use IP addresses and browser fingerprinting to identify people that are logged out or in private browsing mode.
If search results were more anonymous in these states, then we would expect everyoneʼs private browsing mode results to be similar. Thatʼs not what we saw.
To test this more rigorously, we took the organic results, excluding ads and infoboxes, and:
1. Assigned each domain a letter (e.g. A for nytimes.com, B for wsj.com, etc.).
2. Made a string of letters for each personʼs results, e.g. ABDFJKMSL.
3. Compared these strings to see how similar they were to each other.
To do this comparison we counted domain changes between different sets of search results, reducing the differences to a number. For example, ABC -> ACB is one change. (Technically, we used a letter to represent each domain within each search result and calculated the Damerau-Levenshtein edit distance between them.)
Visualization showing how edit distances are calculated to measure the difference between strings.
We saw that when randomly comparing peopleʼs private modes to each other, there was more than double the variation than when comparing someoneʼs private mode to their normal mode:
\* Average of normal and private browsing mode (same user): 1.03 \* Average of private browsing mode (random user): 2.89 \* Average of private browsing mode (five closest users): 2.65
\* Average of normal and private browsing mode (same user): 1.38 \* Average of private browsing mode (random user): 3.28 \* Average of private browsing mode (five closest users): 2.80
\* Average of normal and private browsing mode (same user): 2.23 \* Average of private browsing mode (random user): 4.97 \* Average of private browsing mode (five closest users): 4.25
Visualization showing that thereʼs little difference in results between searching in normal mode and private browsing mode.
We often hear of confusion that private browsing mode enables anonymity on the web, but this finding demonstrates that Google tailors search results regardless of browsing mode. People should not be lulled into a false sense of security that so-called "incognito" mode makes them anonymous.
Study Data and Code
The data is available for download in two parts: Basic non-identifiable participant data, and raw data from the search results.
The code that we wrote to analyze the data is open source and available on our GitHub repository.
For more privacy advice, follow us on Twitter & get our privacy crash course.
HackerNewsBot debug: Calculated post rank: 136 - Loop: 317 - Rank min: 100 - Author rank: 900
Was redirected to here from uBlockOrigin/uAssets#3780 URL(s) where the issue occurs superliitto.fi, sak.fi, akt.fi, selry.fi, akava.fi, oaj.fi, stthl.fi, proliitto.fi, pam.fi, sahkoliitto.fi Descri...
Article word count: 43
HN Discussion: https://news.ycombinator.com/item?id=18298342
Posted by Maakuth (karma: 2938)
Post stats: Points: 118 - Comments: 25 - 2018-10-25T03:16:40Z
\#HackerNews #adblock #filter #finland #list #modified #politically #rule
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.
HackerNewsBot debug: Calculated post rank: 87 - Loop: 137 - Rank min: 80 - Author rank: 86