The role of scientific conferences in R&D

In this post I'm dealing with a very important question from the perspective of a person managing or financing R&D, how does one know how well is R&D performing? If your thought was that you'll measure it by economic success of a product that uses the results of R&D then you are on a wrong track. Namely, the product can be success or a failure because of a number of reasons, of which R&D is only one. So, another way has to be used, and actually this question is very hard. In this post I'll try to point you to a possible solution along with some of its negative sides. Before continuing, just to reiterate that this post is from the perspective of a person managing or financing R&D.

The best possible solution would be that you absolutely trust all your researchers and that they produce only the best results. But this is idealistic case, namely there are no perfect researchers, and even the best ones could produce mediocre results if they are under sufficiently high pressure. So, some form of quality assurance is necessary.

The next best solution would be for you to check what every researcher did and evaluate it by yourself, after all, whom do you trust more than yourself? But this approach also has problems, and not the small ones:

1. When good researchers does something, the only way to track him would be to do the same things he does, and that means doing his job. 
2. Even if you would know so much to be able to analyze how someone does his or her job, that wouldn't scale.
3. Finally, people tend to hate micromanagement, and this would be micromanagement.

So, this approach also wouldn't work. Another approach would be to assign for each researcher another person that would check his work. But this has almost the same problems as if you are doing everything by yourself. Especially problematic could be potential collusion between researchers, i.e. one praises other's work knowing that his own work will be reviewed, too. So, reviewers might have incentive to praise each other's work.

Thus, it is necessary to have review, but the point of the review is to be independent, done by an expert that knows the topic being reviewed and trying to be as objective as possible. You can pay independent researchers for doing review, but that's not done. What's done instead is sending papers to scientific conferences and journals where they are reviewed before being published. The review process is such that the authors don't know who reviewed their paper (blind review) or even reviewers don't know who's paper they are reviewing (double blind review). Before being published in a journal or on a conference, papers have to pass review process and authors are notified about the decision along with receiving reviewers' comments.

So, there is a way you can receive feedback about the work done by your researchers by sending them to conferences or requiring them to publish in journals. But there are additional benefits as well:

1. Even if your researches have the best intention of producing top class results, it is good to have a feedback. In the reviews there could be suggestions on how to improve the work.
2. By participating on conferences your researchers build their professional network from people doing the same or similar things and that might be very helpful on the long run.
3. You should not forget marketing aspects of scientific publications. Namely, this makes you and your people known as an organization that does research and supports their researchers which might attract new researchers and employees.

Many companies having serious R&D do publish on scientific conferences and in journals and they put on their Web pages lists of published works, here are some:

• Hitachi R&D publications
• Google Research Publications
• Microsoft Research Publications
• Facebook Research Publications
• IBM Research Publications (Haifa Lab)
• HP Enterprise Research Publications
• Sony Research Publications (Paris Laboratory)

There are many others, and I might add more to the list later.

One very important thing before I continue. People tend to think that I say that publications are mean and a goal and thus are opposing to the idea of publishing on a scientific conferences. But that's not true. Publications are only a side-product of a work who's goal is to produce something new that could be used to improve company's products!

But, nothing is perfect and so this approach has some issues you have to be aware of:

1. There are a huge number of conferences in the world many of which are at best average. You should strive to go to the best ones because there you'll receive the best feedback and also meet people that are more likely to be researching things that interest you. Which conferences are those depends on the specific research area and you have to search for them, but as a general rule of thumb the lower acceptance ratio, the better conference.
2. As I've said, the papers are only a side-product of the actual work done. But, if too great emphasize is put on conference/journal publication, then researchers start to optimize that criteria instead of doing a good work.
3. You should be careful what you publish in the papers. The moment its published, effectively it's a public knowledge. This is very good from the society perspective, but it might not be so good from the perspective of a company.
4. Publication on the conference is not so cheap. You have to pay conference fee, travel and accommodation expenses, and maybe few more things. This builds up very quickly.
5. Publication in a journal might cost nothing, but it can take time, up to 18 months. The review process for conferences is several months at most.

But in any case, I think that companies should publish as much as possible on a good conferences or in good journals as it has more benefits than drawbacks.

SCADA/ICS security conferences in 2016

On SCADASEC mailing list there was a question about security conferences in 2016 worth attending. I find this question very interesting so I decided to list here all responses received in this thread. The result is shown in the following table:

List of ICS/SCADA conferences in 2016

Important dates	Conference name	Venue	Comment
Conference date: January 12-14, 2016	S4x16 Week	Miami South Beach	Probably the best in the US for a heavy research focus, Dale and team do an excellent job on trying to do a "what's next" approach usually as well as a lot of flash/flair (fun time)
Conference date: February 7-11, 2016	Kaspersky Security Analyst Summit	Tenerife, Spain
Conference date: February 9-11, 2016	DistribuTech	Orange County Convention Center, West Halls A3-4 & B, Orlando, FL	Focused on power grid, very OT centric and not security focused gives a unique look at broader industry
Conference date: February 16-23, 2016	ICS Security Summit	Orlando, FL	Great 2 day conference with opportunity to take training classes if you want, hands-on challenges, live demos, and ~200 strong IT/OT mixed audience
Conference date: April 26-28, 2016	ICS Cyber Security	London, United Kingdom
Conference date: May 3-5, 2016	ICSJWG 2016 Spring Meeting	Scottsdale, AZ	Multiple times a year in different locations): Definitely one to go for anyone new to the ICS community, it's free and the ICSCERT folks are always very kind/professional/awesome.
Conference date: May 30, 2016	ACM Cyber-Physical System Security Workshop (CPSS 2016)	Xi’an, China
Conference date: October 25-27, 2016	4SICS	Stockholm, Sweden	Great IT/OT mix (50% practitioners this past year) with a very similar vibe to S4 in terms of flair/research
Conference date: November 10-11, 2016	11th Annual API Cybersecurity Conference & Expo	Westin Houston Memorial City Houston, Texas	Focused on Oil/Gas, very diverse group of speakers with a lot of vendor interaction. Note: I'm extrapolating in this case as when and where the next conference will take place.

The column titled Comment is taken from this mail message, so I'm crediting the original author as I don't have any first-person experience with the mentioned conferences. Also, you can find additional list of conferences here and here.

CFP: MIPRO ISS

Starting from this year I'm going to be a vice chair of Information Systems Security event that is a part of a larger MIPRO conference. The reason I took this role is that I believe that relevant security event is missing in this region and that this conference (I'll say conference not event from now on and by that I'll refer to ISS event) can fill the void. Furthermore, I believe there is lot of a room for improvements, which is of course mandatory if this conference is to become regional, and I have some ideas what and how to do it. But it will take me some time until I articulate what I intend to do. In the meantime, CFP was published [PDF].

I don't find conferences appropriate for publishing finished work, journals a better for that purpose. Conferences are, on the other hand, ideal for presenting your work in progress in order to solicit feedback so that, in the end, you improve quality of your research. I especially invite students, undergraduate, graduate and postgraduate, so submit their work for diploma thesis or PhDs. Also of great interest are findings of weakness (vulnerabilities) somewhere, I invite you to present your finding on the conference. Of course, in that case you should be careful first to notify those you found a vulnerability so that they have time to react.

Problem with authors that do not attend conferences and pay fees...

Yesterday I had an idea and before I forget about it, I thought it could be good to share it with the world. :) The reason I got that idea is that, very likely, I'm going to be vice chair of Information Systems Security event.

As far as I know, there are lot of authors that submit their papers to conferences with the intention to be published in proceedings, but they don't show up on the conference to present their work. This is regarded as rude. But, even worse is that there are also those that don't pay conference fees. Namely, usually the proceedings have to be sent to the press before deadline for payments, and also, there are those people that pay on the spot, which is also acceptable.

So, the problem is that authors promise that they will pay and attend conference, but in the end they don't do neither of those two things. So, the question is, how to revoke papers of the authors that didn't pay? Obviously, it is not possible to revoke paper from published proceedings. Nor it is an option to print proceedings later because it would incur additional costs (shipping). Lately, it is also common that conferences publish CDs with proceedings only. Those potentially could be duplicated on spot, but then again you have a problem that you don't know until the last day of the conference who will or who will not attend/pay and thus you would need to postpone CD until the end. This is also unacceptable as some people come only for a day or two and they leave before the last day of the conference, so again you have shipping costs. But shipping is not the only problem, the other is that people that attend the conference like to have papers in order to more easily follow presentation.

The proposed solution is simple. First of all, the proceedings would be published only on CD or USB. That is environmentally friendly approach. Next, all the accepted papers would be placed on the CD. BUT, they are encrypted and inaccessible without a key that is NOT on the CD/USB itself. Each paper with its own, unique, key. That key would be published on Web pages of conference (or on IEEE/ACM pages). Obviously, only those that payed (and attended the conference) will have keys published and thus, their papers will be part of the proceedings. The others will be completely unavailable.

This can be made quite transparent, in a sense that some application is started that obtains keys, and stores it locally if necessary so that content can be read when offline.

I think that's good idea. Even though its realization is questionable. What do you think? :)